Staggered motion compensation for preprocessing video with overlapped 3d transforms

ABSTRACT

In one method embodiment, receiving at a frame matching module a first frame comprising first plural blocks and plural frames that each comprise a plurality of blocks to be matched to the first plural blocks of the first frame, the first plural blocks and the plurality of blocks comprising luma blocks; for each of the frame pair matchings, selecting one border configuration among a plurality of border configurations, the border configuration selected for the each of the frame pair matchings unique; appending a border of pixels to the frames of the each of the frame pair matchings based on the selected border configuration; and block matching the first plural blocks with the plurality of blocks.

TECHNICAL FIELD

The present disclosure relates generally to video noise reduction.

BACKGROUND

Filtering of noise in video sequences is often performed to obtain asclose to a noise-free signal as possible. Spatial filtering requiresonly the current frame (e.g., picture) to be filtered and notsurrounding frames in time. Spatial filters, when implemented withouttemporal filtering, may suffer from blurring of edges and detail. Forthis reason and the fact that video tends to be more redundant in timethan space, temporal filtering is often employed for greater filteringcapability with less visual blurring. Since video contains both staticscenes and objects moving with time, temporal filters for video includemotion compensation from frame to frame for each part of the movingobjects to prevent trailing artifacts of the filtering.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the disclosure can be better understood with referenceto the following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the present disclosure. Moreover, in thedrawings, like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a block diagram that illustrates an example environment inwhich certain embodiments of preprocessing sub-systems and methods ofvideo denoising (VDN) systems and methods can be implemented.

FIGS. 2A-2C are schematic diagrams that conceptually illustrateprocessing implemented by various example embodiments of VDN systems andmethods.

FIG. 3 is a block diagram that illustrates one example VDN systemembodiment comprising frame alignment and overlapped block processingmodules and respectively associated preprocessing sub-systems.

FIG. 4 is a block diagram that illustrates one example embodiment of aframe alignment module.

FIG. 5 is a block diagram that illustrates one example embodiment of aframe matching module, comprising example preprocessing sub-systems, ofa frame alignment module.

FIG. 6A is a block diagram that illustrates example luma border pixelconfigurations applied by an embodiment of a staggered motioncompensation sub-system residing in an example frame matching module.

FIG. 6B is a block diagram that illustrates an example constellation ofa border configuration resulting from staggered motion compensationprocessing.

FIG. 7 is a block diagram that illustrates an overlapped superblockapplied to an example field-frame map as implemented by a field-frameprocessing sub-system.

FIGS. 8A-8D are block diagrams that illustrate a modifiedone-dimensional (1D) transform used in an overlapped block processingmodule, the 1D transform illustrated with progressively reducedcomplexity.

FIG. 9 is a block diagram that conceptually illustrates the use oftemporal modes in an overlapped block processing module.

FIG. 10 is a block diagram that illustrates an example mechanism forthresholding.

FIG. 11 is a block diagram that illustrates an example scene changedetection mask utilized for detecting scene changes between matchedframe pairs.

FIG. 12A is a flow diagram that illustrates an example method embodimentimplemented by an embodiment of a field-frame processing sub-system inconjunction with an overlapped block processing module.

FIG. 12B is a flow diagram that illustrates another example methodembodiment implemented by an embodiment of a field-frame processingsub-system in conjunction with an overlapped block processing module.

FIG. 13 is a flow diagram that illustrates an example method embodimentimplemented by an embodiment of a scene change detection sub-system.

FIG. 14 is a flow diagram that illustrates an example method embodimentimplemented by an embodiment of a staggered motion compensationsub-system.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

In one method embodiment, receiving at a frame matching module a firstframe comprising first plural blocks and plural frames that eachcomprise a plurality of blocks to be matched to the first plural blocksof the first frame, the first plural blocks and the plurality of blockscomprising luma blocks; for each of the frame pair matchings, selectingone border configuration among a plurality of border configurations, theborder configuration selected for the each of the frame pair matchingsunique; appending a border of pixels to the frames of the each of theframe pair matchings based on the selected border configuration; andblock matching the first plural blocks with the plurality of blocks.

Example Embodiments

Disclosed herein are various example embodiments of preprocessingsub-systems and methods of a video denoising (VDN) system. Oneembodiment of a preprocessing sub-system, referred to herein as afield-frame processing (FFP) sub-system, comprises logic thatsimultaneously processes two interleaved n×n (e.g., 8×8) pixelsub-blocks of an n×m (e.g., 8×16) pixel superblock, where n and m arenon-negative integer numbers. For a frame, a superblock is referred toherein as two vertically adjacent blocks (top/bottom), and for a field,a superblock is referred to herein as two interleaved blocks having avertical span of two adjacent blocks. For purposes of brevity and forillustration only, each example luma superblock is described as havingan 8×16 size, with the understanding that superblocks of other sizes(e.g., 16×32, etc.) as well as different-sized sub-blocks (i.e., the twoblocks that make up a superblock) of a superblock are contemplated to bewithin the scope of the disclosure. The simultaneous, coupled processingfor interlaced video as interleaved fields (e.g., single channelprocessing) as disclosed herein is in contrast to decoupled processingas two separate fields (e.g., splitting the two fields of frame picturesinto separate channels), hence resulting in some implementations inreduced complexity. In one embodiment, the FFP sub-system provides(e.g., once for a reference frame) a field-frame map by logicallypartitioning a block matched frame into plural non-overlappingsuperblocks, the block matched frame destined to be denoised via anoverlapped block process. For each superblock in the field-frame map,the FFP sub-system provides a respective frame or field designation.Based on the field-frame map and the coordinate position of anoverlapped block (e.g., overlapped superblock) that is subject tocomparison with the superblocks of the field-frame map, the FFPsub-system further determines whether the sub-blocks of the overlappedblock at the particular coordinate position are to each undergooverlapped block processing (e.g., video denoising) as a frame or field.

Another embodiment of a preprocessing sub-system, referred to herein asa scene change detection (SCD) sub-system, comprises logic that providesa mechanism to detect scene changes and that further cooperates withmode decision logic that handles the detected scene change in one ormore temporal depth modes. In one embodiment, the SCD sub-system isimplemented during a block matching stage involving iterative respectivesets of noise-filtered blocks from two full-size (e.g., not decimated)frames. As the iteration of block matching between the sets progresses,a decision is made, block-by-block, as to the closeness of the matchuntil it is determined that a scene change detection has been triggered.In one embodiment, a bit mask is used to track the location of adetected scene change among a window of frames to facilitateimplementation of a suitable temporal mode.

Yet another embodiment of a preprocessing sub-system, referred to hereinas a staggered motion compensation (SMC) sub-system, comprises logicthat applies a varying border configuration of pixels around a referenceframe, the variation occurring between different pairings of thereference frame with another frame (and further difference in borderconfigurations per fields of the same reference frame) in an intendedblock matching operation. In other words, each time a frame is matchedto the reference frame, the same border configuration is used betweenthe reference frame and the other frame, yet a different border is usedfor each pairing of the reference frame with another frame (and thenremoved at the completion of frame matching). The SMC sub-systemeffectively positions the seams of the n×m blocks for luma in differentplaces each time a frame match is performed, eliminating or mitigatingfrom the denoised frames blocking or other artifacts resulting fromaggregated seams.

As indicated above, the various preprocessing sub-system embodiments aredescribed in the context of a VDN system. In one embodiment, the VDNsystem comprises a frame alignment module and an overlapped blockprocessing module, the overlapped block processing module configured todenoise video in a three-dimensional (3D) transform domain using motioncompensated overlapped 3D transforms. In particular, certain embodimentsof VDN systems motion compensate a set of frames surrounding a currentframe, and denoise the frame using 3D spatio-temporal transforms withthresholding of the 2D and/or 3D transform coefficients. One or more VDNsystem embodiments provide several advantages or distinctive featuresover brute force methods of conventional systems, includingsignificantly reduced computational complexity that enableimplementation in real-time silicon (e.g., applicable to real-timeapplications, such as pre-processing of frames for real-timebroadcasting of encoded pictures of a video stream), such asnon-programmable or programmable hardware including field programmablegate arrays (FPGAs), and/or other such computing devices. Severaladditional distinctive features and/or advantages of VDN systemembodiments, explained further below, include the decoupling of blockmatching and inverse block matching from an overlapped block processingloop, reduction of accumulation buffers from 3D to 2D+n (where n is aninteger number of accumulated frames less than the number of frames in a3D buffer), and the “collapsing” of frames (e.g., taking advantage ofthe fact that neighboring frames have been previously frame matched toreduce the amount of frames entering the overlapped block processingloop while obtaining the benefit of the information from the full scopeof frames from which the reduction occurred for purposes of denoising).Such features and/or advantages enable substantially reduced complexityblock matching. Further distinctive features include, among others, acustomized-temporal transform and temporal depth mode selection, alsoexplained further below.

These advantages and/or features, among others, are describedhereinafter in the context of an example subscriber television networkenvironment, with the understanding that other video environments mayalso benefit from certain embodiments of the preprocessing sub-systemsand VDN systems and methods and hence are contemplated to be within thescope of the disclosure. It should be understood by one having ordinaryskill in the art that, though specifics for one or more embodiments aredisclosed herein, such specifics as described are not necessarily partof every embodiment.

FIG. 1 is a block diagram of an example environment, a subscribertelevision network 100, in which certain embodiments of preprocessingsub-systems of a VDN system may be implemented. The subscribertelevision network 100 may include a plurality of individual networks,such as a wireless network and/or a wired network, including wide-areanetworks (WANs), local area networks (LANs), among others. Thesubscriber television network 100 includes a headend 110 that receives(and/or generates) video content, audio content, and/or other content(e.g., data) sourced at least in part from one or more serviceproviders, processes and/or stores the content, and delivers the contentover a communication medium 116 to one or more client devices 118through 120. The headend 110 comprises an encoder 114 having videocompression functionality, and a pre-processor or VDN system 200configured to receive a raw video sequence (e.g., uncompressed videoframes or pictures), at least a portion of which (or the entirety) iscorrupted by noise. Such noise may be introduced via camera sensors,from previously encoded frames (e.g., artifacts introduced by a priorencoding process from which the raw video was borne, among othersources). The VDN system 200 is configured to denoise each picture orframe of the video sequence and provide the denoised pictures or framesto the encoder 114, enabling, among other benefits, the encoder toencode fewer bits than if noisy frames were inputted to the encoder. Insome embodiments, at least a portion of the raw video sequence maybypass the VDN system 200 and be fed directly into the encoder 114. TheVDN system 200 further comprises a preprocessing sub-system 300, thepre-processing sub-system including, among other components orsub-systems described below, one or a combination of a field-frameprocessing (FFP) sub-system, scene change detection (SCD) sub-system,and/or staged motion compensation (SMC) sub-system as explained furtherbelow.

Throughout the disclosure, the terms pictures and frames are usedinterchangeably. In some embodiments, the uncompressed video sequencesmay be received in digitized format, and in some embodiments,digitization may be performed in the VDN system 200. In someembodiments, the VDN system 200 may comprise a component that may bephysically and/or readily de-coupled from the encoder 114 (e.g., such asin the form of a plug-in-card that fits in a slot or receptacle of theencoder 114). In some embodiments, the VDN system 200 may be integratedin the encoder 114 (e.g., such as integrated in an applications specificintegrated circuit or ASIC). Although described herein as apre-processor to a headend component or device, in some embodiments, theVDN system 200 may be co-located with encoding logic at a client device,such as client device 118, or positioned elsewhere within a network,such as at a hub or gateway.

The headend 110 may also comprise other components, such as QAMmodulators, routers, bridges, Internet Service Provider (ISP) facilityservers, private servers, on-demand servers, multi-media messagingservers, program guide servers, gateways, multiplexers, and/ortransmitters, among other equipment, components, and/or deviceswell-known to those having ordinary skill in the art. Communication ofInternet Protocol (IP) packets between the client devices 118 through120 and the headend 110 may be implemented according to one or more of aplurality of different protocols, such as user datagram protocol(UDP)/IP, transmission control protocol (TCP)/IP, among others.

In one embodiment, the client devices 118 through 120 comprise set-topboxes coupled to, or integrated with, a display device (e.g.,television, computer monitor, etc.) or other communication devices andfurther coupled to the communication medium 116 (e.g., hybrid-fibercoaxial (HFC) medium, coaxial, optical, twisted pair, etc.) via a wiredconnection (e.g., via coax from a tap) or wireless connection (e.g.,satellite). In some embodiments, communication between the headend 110and the client devices 118 through 120 comprises bi-directionalcommunication over the same transmission medium 116 by which content isreceived from the headend 110, or via a separate connection (e.g.,telephone connection). In some embodiments, communication medium 116 maycomprise of a wired medium, wireless medium, or a combination ofwireless and wired media, including by way of non-limiting exampleEthernet, token ring, private or proprietary networks, among others.Client devices 118 through 120 may henceforth comprise one of manydevices, such as cellular phones, personal digital assistants (PDAs),computer devices or systems such as laptops, personal computers, set-topterminals, televisions with communication capabilities, DVD/CDrecorders, among others. Other networks are contemplated to be withinthe scope of the disclosure, including networks that use packetsincorporated with and/or compliant to other transport protocols orstandards.

The VDN system 200 (and associated sub-systems, such as thepreprocessing sub-system 300) may be implemented in hardware, software,firmware, or a combination thereof (collectively or individually alsoreferred to herein as logic). To the extent certain embodiments of theVDN system 200 or a portion thereof are implemented in software orfirmware, executable instructions or code for performing one or moretasks of the VDN system 200 are stored in memory or any other suitablecomputer readable medium and executed by a suitable instructionexecution system. In the context of this document, a computer readablemedium is an electronic, magnetic, optical, or other physical device ormeans that can contain or store a computer program for use by or inconnection with a computer related system or method.

To the extent certain embodiments of the VDN system 200 or a portionthereof are implemented in hardware, the VDN system 200 may beimplemented with any or a combination of the following technologies,which are all well known in the art: a discrete logic circuit(s) havinglogic gates for implementing logic functions upon data signals, anapplication specific integrated circuit (ASIC) having appropriatecombinational logic gates, programmable hardware such as a programmablegate array(s) (PGA), a field programmable gate array (FPGA), etc.

Having described an example environment in which the VDN system 200 maybe employed, attention is directed to FIGS. 2A-2C, which compriseschematic diagrams that conceptually illustrate data flows and/orprocessing implemented by various example embodiments of VDN systems andmethods. Progressing from FIG. 2A to FIG. 2B and then to FIG. 2Crepresents a reduction in processing complexity, and hencelike-processing throughout these three figures are denoted with the samenumerical reference and an alphabetical or alphanumeric suffix (e.g., a,b, and c, or a−1, etc.) that may change from each figure for a givencomponent or diagram depending on whether there is a change or reductionin complexity to the component or represented system 200. Further, each“F” (e.g., F0, F1, etc.) shown above component 220 a in FIG. 2A (andlikewise shown and described in association with other figures) is usedto denote frames that have not yet been matched to the reference frame(F4), and each “M” (e.g., M0, M1, etc.) is used to denote frames thathave been matched (e.g., to the reference frame). Note that use of theterm “component” with respect to FIGS. 2A-2C does not imply thatprocessing is limited to a single electronic component, or that each“component” illustrated in these figures are necessarily separateentities. Instead, the term “component” in these figures graphicallyillustrates a given process implemented in the VDN system embodiments,and is used instead of “block,” for instance, to avoid confusion in theterm, block, when used to describe an image or pixel block.

Overall, VDN system embodiment, denoted 200 a in FIG. 2A, can besubdivided into frame matching 210 a, overlap block processing 250 a,and post processing 270 a. In frame matching 210 a, entire frames arematched at one time (e.g., a single interval of time or singleprocessing stage), and hence do not need to be matched duringimplementation of overlapped block processing 250 a. In other words,block matching in the frame matching process 210 a is decoupled (e.g.,blocks are matched without block overlapping in the frame matchingprocess) from overlapped block processing 250 a, and hence entire framesare matched and completed for a given video sequence before overlappedblock processing 250 a is commenced for the given video sequence. Bydecoupling frame matching 210 a from overlapped block processing 250 a,block matching is reduced by a factor of sixty-four (64) when overlappedblock processing 250 a has a step size of s=1 pixel in both the verticaland horizontal direction (e.g., when compared to integrating overlappedblock processing 250 a with the frame matching 210 a). If the step sizeis s=2, then block matching is reduced by a factor of sixteen (16). Onehaving ordinary skill in the art should understand that various stepsizes are contemplated to be within the scope of the disclosure, theselection of which is based on factors such as available computationalresources and video processing performance (e.g., based on evaluation ofPSNR, etc.).

Shown in component 220 a are eight (8) inputted contiguous frames, F0(t)through F7(t) (denoted above each symbolic frame as F0, F1, etc.). Theeight (8) contiguous frames correspond to a received raw video sequenceof plural frames. In other words, the eight (8) contiguous framescorrespond to a temporal sequence of frames. For instance, the frames ofthe raw video sequence are arranged in presentation output order (whichmay be different than the transmission order of the compressed versionsof these frames at the output of the headend 110). In some embodiments,different arrangements of frames and/or different applications arecontemplated to be within the scope of the disclosure. Note thatquantities fewer or greater than eight frames may be used in someembodiments at the inception of processing. Frame matching is symbolizedin FIGS. 2A-2C by the arrow head lines, such as represented in component220 a (e.g., from F0 to F4, etc.).

As illustrated by the arrow head lines, frames F0(t) through F3(t) arematched to F4(t), meaning that blocks (e.g., blocks of pixels or imageblocks, such as 8×8, 8×4, etc.) have been selected from those frameswhich most closely match blocks in F4(t) through a motionestimation/motion compensation process as explained below. The result offrame matching is a set of frames M0(t) through M7(t), whereM4(t)=F4(t), as shown in component 230 a. Frames M0(t) through M7(t) areall estimates of F4(t), with M4(t)=F4(t) being a perfect match. M4(t)and F4(t) are used interchangeably herein.

Overlapped block processing 250 a is symbolically represented withcomponents 252 (also referred to herein as a group of matched noisyblocks or similar), 254 (also referred to herein as 3D denoising orsimilar), 256 (also referred to as a group or set of denoised blocks orsimilar), and 260 a (also referred to herein as pixel accumulationbuffer(s)). Overlapped block processing 250 a moves to a pixel locationi,j in each of the matched frames (e.g., the same, co-located, or commonpixel location), and for each loop, takes in an 8×8 noisy block b(i,j,t)(e.g., 240 a) from each matched frame M0 through M7, with the top leftcorner at pixel position i,j, so that b(i,j,t)=Mt(i:i+7, j:j+7). Notethat i,j are vertical and horizontal indices which vary over the entireframe, indicating the position in the overlapped processing loop. Forinstance, for a step size s=1, i,j takes on every pixel position in theframe (excluding boundaries of 7 pixels). For a step size s=2 i,j takeson every other pixel. Further note that 8×8 is used as an example blocksize, with the understanding that other block sizes may be used in someembodiments of overlapped block processing 250 a. The group of eight (8)noisy 8×8 blocks (252) is also denoted as b(i,j, 0:7). Note that theeight (8) noisy blocks b(i,j, 0:7) (252) are all taken from the samepixel position i,j in the matched frames, since frame alignment (as partof frame matching processing 210 a) is accomplished previously. Framematching 210 a is decoupled from overlapped block processing 250 a.

3D denoising 254 comprises forward and inverse transforming (e.g., 2Dfollowed by 1D) and thresholding (e.g., 1D and/or 2D), as explainedfurther below. In general, in 3D denoising 254, a 2D transform isperformed on each of the 8×8 noisy blocks (252), followed by a 1Dtransform across the 2D transformed blocks. After thresholding, theresult is inverse transformed (e.g., 1D, then 2D) back to pixel blocks.The result is a set of eight (8) denoised blocks bd(i,j, 0:7) (256).

For each loop, there are eight (8) denoised blocks (256), but in someembodiments, not all of the blocks of bd(i,j, 0:7) are accumulated tothe pixel accumulation buffers 260 a, as symbolically represented by theframes and blocks residing therein in phantom (dashed lines). Rather,the accumulation buffers 260 a comprise what is also referred to hereinas 2D+c accumulation buffers 260 a, where c represents an integer valuecorresponding to the number of buffers for corresponding frames ofdenoised blocks in addition to the buffer for A4. A 2D accumulationbuffer corresponds to only A4 (the reference frame) being accumulatedusing bd(i,j,4) (e.g., denoised blocks bd(i,j,4) corresponding to frameA4 are accumulated). In this example, another buffer corresponding toc=1 is shown as being accumulated, where the c=1 buffer corresponds todenoised blocks bd(i,j,7) corresponding to frame AM7.

It follows that for an eight (8) frame window, a 2D+7 accumulationbuffer equals a 3D accumulation buffer. Further, it is noted that usinga 2D+1 accumulation buffer is analogous in the time dimension to using astep size s=4 in the spatial dimension (i.e. the accumulation isdecimated in time). Accordingly, c can be varied (e.g., from 0 to adefined integer value) based on the desired visual performance and/oravailable computational resources. However, in some embodiments, a 3Daccumulation buffer comprising denoised pixels from plural overlappedblocks is accumulated for all frames.

In overlapped block processing 250 a, blocks bd(i,j,4) and bd(i,j,7) areaccumulated in accumulation buffers 260 a at pixel positions i,j sincethe accumulation is performed in the matched-frame domain, circumventingany need for inverse block matching within the overlapped blockprocessing loop 250 a. Further, uniform weighting (e.g., w(i,j)=1 or noweighting at all) for all denoised blocks is implemented, whichsignificantly reduces complexity. Note that in some embodiments,non-uniform weighting may be implemented. Note that in some embodiments,buffers for more than two frames (e.g., c>1) may be implemented. For a2D+1 buffer, a frame begins denoising when it becomes A7, since there isan accumulation buffer for A7 for the 2D+1 accumulation buffer 260 a. A7receives a second (2^(nd)) iteration of denoising when it becomes A4.The two are merged as shown in post-processing 270 a, as explainedfurther below.

From the accumulation buffers 260 a, post processing 270 a isimplemented, which comprises in one embodiment the processes of inverseframe matching 272 (e.g., as implemented in an inverse frame matchingmodule or logic), delay 274 (e.g., as implemented in a delay module orlogic), and merge and normalizing 276 (e.g., as implemented in merge andnormalize module or logic). Since in one embodiment the accumulationbuffer corresponding to AM7 is in the matched frame domain (e.g., frame7 matched to frame 4), after the overlapped block processing iscompleted, data flow advances to inverse frame matching 272 to inverseframe match AM7(t) to obtain A7(t). As noted, this operation occurs onceoutside of overlapped block processing 250 a. A7(t) is then delayed(274), in this example, three frames, and merged (added) and normalized276 to A4(t) (as represented by the dotted line arrowhead) to outputFD4(t), the denoised frame. Had the inverse frame matching 272 beenimplemented in the overlapped block processing 250 a, the inverse framematching would move a factor of sixty-four (64) more blocks than theimplementation shown for a step size s=1, or a factor of 16 more fors=2.

Ultimately, after the respective blocks for each accumulated frame havebeen accumulated from plural iterations of the overlapped blockprocessing 250 a, the denoised and processed frame FD4 is output to theencoder or other processing devices in some embodiments. As explainedfurther below, a time shift is imposed in the sequence of framescorresponding to frame matching 210 a whereby one frame (e.g., F0) isremoved and an additional frame (not shown) is added for frame matching210 a and subsequent denoising according to a second or subsequenttemporal frame sequence or temporal sequence (the first temporalsequence associated with the first eight (8) frames (F0-F7) discussed inthis example). Accordingly, after one iteration of frame processing(e.g., frame matching 210 a plus repeated iterations or loops ofoverlapped block processing 250 a), as illustrated in the example ofFIG. 2A, FD4(t) is output as a denoised version of F4(t). As indicatedabove, all of the frames F0(t) through F7(t) shift one frame (alsoreferred to herein as time-shifted) so that F0(t+1)=F1(t),F1(t+1)=F2(t), etc., and a new frame F7(t+1) (not shown) enters the“window” (component 220 a) of frame matching 210 a. Note that in someembodiments, greater numbers of shifts can be implemented to arrive atthe next temporal sequence. Further, F0(t) is no longer needed at t+1 soone frame leaves the window (e.g., the quantity of frames outlined incomponent 220 a). For the 8-frame case, as one non-limiting example,there is a startup delay of eight (8) frames, and since three (3) futureframes are needed to denoise F4(t) and F5 through F7, there is a generaldelay of three (3) frames.

Referring now to FIG. 2B, shown is a VDN system embodiment, denoted 200b, with further reduced computational complexity compared to the VDNsystem embodiment 200 a illustrated in FIG. 2A. The simplification inFIG. 2B is at least partially the result of the 2D+1 accumulationbuffers 260 a and a modified 1D temporal transform, as explained furtherbelow. In the above-description of the 2D+1 accumulation buffers 260 ain FIG. 2A, it is noted that the 2D+1 accumulation buffers 260 a requireonly buffers (e.g., two) for denoised blocks, bd(i,j,4) and bd(i,j,7).Accordingly, a further reduction in complexity includes the “collapse”of the left four (4) frames, F0 through F3, into a singlesummation-frame FSUM, and frame-matching the summation-frame to F4, asillustrated in frame matching 210 b, and in particular, component 220 bof FIG. 2B. The collapse to a single summation frame comprises anoperation which represents a close approximation to the left-hand sideframe matching 210 a illustrated in FIG. 2A. Since F0 through F3 werepreviously matched together at time t−4, no matching operations areneeded on those individual frames. Instead, frame matching 210 b matchesthe sum, FSUM, from time t−4 to F4, where

$\begin{matrix}\begin{matrix}{{{FSUM}\left( {t - 4} \right)} = {\sum\limits_{j = 4}^{7}{{{Mj}\left( {t - 4} \right)}{{FSUM}\left( {t - 4} \right)}}}} \\{= {\sum\limits_{j = 4}^{7}{{{Mj}\left( {t - 4} \right)}.}}}\end{matrix} & {{Eq}.\mspace{14mu} (1)}\end{matrix}$

Eq. (1) implies that FSUM(t) is the sum of the four (4) frame-matched F4through F7 frames, denoted M4(t) through M7(t) at time t, and is usedfour (4) frames (t−4) later as the contribution to the left (earliest)four (4) frames in the 3D transforms. A similar type of simplificationis made with respect to frames F5 and F6. After the four (4) frames F0through F3 are reduced to one FSUM frame, and F5 and F6 are reduced to asingle frame, and then matched to F4(t), there are only four (4) matchedframes, MSUM (actually MSUM0123), M4, M5+6, and M7, as noted incomponent 230 b, and therefore the entire overlapped block processing250 b proceeds using just four (4) (b(i,j, 4:7)) matched frames. Thenumber of frames in total that need matching to F4(t) is reduced tothree (3) in the VDN system embodiment 200 b of FIG. 2B from seven (7)in the VDN system embodiment 200 a in FIG. 2A. In other words,overlapped block processing 250 b receives as input the equivalent ofeight (8) frames, hence obtaining the benefit of eight (8) frames usinga fewer number of frame estimates.

FIG. 2C is a block diagram of a VDN system embodiment, 200 c, thatillustrates a further reduction in complexity from the systemembodiments in FIGS. 2A-2B, with particular emphasis at the accumulationbuffers denoted 260 b. In short, the accumulation buffer 260 b comprisesa 2D accumulation buffer, which eliminates the need for inverse motioncompensation and reduces the complexity of the post-accumulatingprocessing 270 b to a normalization block. In this embodiments, theframes do not need to be rearranged into the frame-matching (e.g.,component 220 b) as described above in association with FIG. 2B.Instead, as illustrated in the frame matching 210 c, F7 is matched toF6, and the result is added together to obtain FSUM67. F5 is matched toF4 using the motion vectors from two (2) frames earlier (when F5, F4were positioned in time at F7, F6 respectively), so this manner of framematching is shown as a dotted line between F4 and F5 in component 220 cin FIG. 2C. As before, FSUM0123 represents four (4) frames matched, four(4) frames earlier, and summed together. In summary, frame matching 210c for the 2D accumulation buffer 260 b matches three (3) frames to F4:FSUM0123, FSUM67 and F5.

Having described conceptually example processing of certain embodimentsof VDN systems 200, attention is now directed to FIG. 3, which comprisesa block diagram of VDN system embodiment 200 c-1. It should beunderstood by one having ordinary skill in the art in the context of thepresent disclosure that the various components shown in FIG. 3 maycomprise, in one embodiment, software modules comprising code executedby a processor(s) (e.g., central processing unit or CPU, digital signalprocessor or DSP, etc.) of an FPGA or other device. In some embodiments,one or more of the components may be configured in hardware, or in someembodiments, the VDN system embodiment 200 may be configured as a mix ofhardware and software components. It is noted that the architecture andfunctionality described hereinafter is based on VDN system embodiment200 c described in association with FIG. 2C, with the understanding thatsimilar types of architectures and components for VDN system embodiments200 a and 200 b can be derived by one having ordinary skill in the artbased on the teachings of the present disclosure without undueexperimentation. VDN system embodiment 200 c-1 comprises a framealignment module 310 that further comprises a staggered motioncompensation (SMC) sub-system 300A and a scene change detection (SCD)sub-system 300B. The VDN system embodiment 200 c-1 further comprises afield-frame processing (FFP) sub-system 300C, the FFP sub-systemcomprising a field-frame mapping logic 380 and a field-frame logic 390,each of which are further described below. Note that in someembodiments, functionality for the field-frame mapping logic 380 andfield-frame logic 390 may be combined, or in some embodiments, furtherdistributed among a greater number of modules and/or combined withfunctionality of other modules. The VDN system embodiment 200 c-1further comprises an overlapped block processing module 350 (that in oneembodiment comprises, among other logic, the field-frame logic 390 ofthe field-frame processing sub-system 300C that cooperates with thefield-frame mapping logic 380), an accumulation buffer 360, and anormalization module 370 (post-accumulation processing). It is notedthat frame processing 250 in FIGS. 2A-2C correspond to the processingimplemented by the frame alignment module 310, and overlap blockprocessing 250 corresponds to the processing implemented by theoverlapped block processing module 350. In addition, the 2D accumulationbuffer 360 and the normalization module 370 implement processingcorresponding to the components 260 and 270, respectively, in FIG. 2C.Note that in some embodiments, functionality may be combined into asingle component or distributed among more or different modules.

As shown in FIG. 3, the frame alignment module 310 receives plural videoframes F4(t), F6(t), and F7(t), where F4(t) is the earliest frame intime, and t is the time index which increments with each frame. Theframe FSUM0123(t)=FSUM4567(t−4) represents the first four (4) framesF0(t) through F3(t) (F4(t−4) through F7(t−4)) which have been matched toframe F0(t) (F4(t−4)) previously at time t=t−4. The frame alignmentmodule 310 produces the following frames for processing by theoverlapped block processing module 350, the details of which aredescribed below: M4(t), MSUM67(t), M5(t), and MSUM0123(t).

Before proceeding with the description of the overlapped blockprocessing module 350, an example embodiment of the frame alignmentmodule 310 is explained below and illustrated in FIG. 4. One exampleembodiment of a frame alignment module 310 a, shown in FIG. 4, receivesframes F4(t), F6(t), F7(t), and FSUM0123(t). It is noted that M5(t) isthe same as M76(t−2), which is the same as M54(t). F7(t) isframe-matched to F6(t) at frame match module 402, producing M76(t).After a delay (404) of two (2) frames, M76(t) becomes M54(t), which isthe same as M5(t). Note that blocks labeled “delay” and shown in phantom(dotted lines) are intended to represent delays imposed by a givenoperation, such as access to memory. F6(t) is summed with M76(t) atsummer 406, resulting in FSUM67(t). FSUM67(t) is frame-matched to F4(t)at frame match module 408, producing MSUM67(t). MSUM67(t) is multipliedby two (2) and added to F4(t) and M5(t) at summer 410, producingFSUM4567(t). FSUM4567(t) can be viewed as frames F5(t) through F7(t) allmatched to F4(t) and summed together along with F4(t). FSUM4567(t) isdelayed (412) by four (4) frames producing FSUM0123(t) (i.e.,FSUM4567(t−4)=FSUM0123(t)). FSUM0123(t) is frame-matched to F4(t) atframe match module 414 producing MSUM0123(t). Accordingly, the output ofthe frame alignment module 310 a comprises the following frames:MSUM0123(t), M4(t), M5(t), and MSUM67(t).

One having ordinary skill in the art should understand in the context ofthe present disclosure that equivalent frame processing to thatillustrated in FIG. 4 may be realized by the imposition of differentdelays in the process, and hence use of different time sequences offrames in a given temporal sequence as the input. For instance, an extraframe delay corresponding to the derivation of frames F6(t) and F7(t)from an FSUM87(t) may be inserted (e.g., immediately after summer 406),resulting in frames FSUM67(t), enabling all frame-matching operations towork out of memory.

One example embodiment of a frame match module, such as frame matchmodule 402, is illustrated in FIG. 5. It should be understood that thediscussion and configuration of frame match module 402 similarly appliesto frame match modules 408 and 414, though not necessarily limited toidentical configurations. Frame match module 402 comprises motionestimation (ME) and motion compensation (MC) functionality (alsoreferred to herein as motion estimation logic and motion compensationlogic, respectively), which is further subdivided into luma ME 502 (alsoluma ME logic or the like), chroma ME 504 (also chroma ME logic or thelike), luma MC 506 (also luma MC logic or the like), and chroma MC 508(also chroma MC logic or the like). In one embodiment, the luma ME 502comprises a binomial filter 510 (also referred to herein as a pixelfilter logic), a decimator 512 (also referred to herein as decimatorlogic), a decimated block matching (DECBM) module 514 (also referred toherein as decimated block matching logic), a full pixel bock matching(BM) module 516 (also referred to herein as full pixel block matchinglogic), and a luma refinement BM module 518. The chroma ME 504 comprisesa chroma refinement BM module 520. The luma MC 506 comprises a luma MCmodule 522, and the chroma MC 508 comprises a chroma MC module 524.

The frame matching module 402 takes as input two video frames, each ofwhich includes luminance and chrominance data in either of the wellknown CCIR-601 4:2:0 or 4:2:2 formats, though not limited to theseformats, and in some embodiments may receive a proprietary format amongother types of formats. For 4:2:0 formats, the chrominance includes twochannels subsampled by a factor of two (2) in both the vertical andhorizontal directions. For 4:2:2 formats, the chrominance is subsampledin only the horizontal direction. The luminance inputs are denoted inFIG. 5 as LREF and LF, which represent the reference frame luminance anda frame luminance to match to the reference, respectively. Similarly,the corresponding chrominance inputs are denoted as CREF (reference) andCF (frame to match to the reference). The output of the frame matchprocess is a frame which includes luminance (LMAT) data and chrominance(CMAT) data. For instance, according to the embodiments described inassociation with, and illustrated in, FIGS. 2C, 3A, and 5, LREF, CREF,LF, CF and LMAT, CMAT correspond to the sets of frames given in Table 1below:

TABLE 1 Sets of Frames Undergoing Frame Matching LREF, LMAT, CREF LF, CFCMAT Description F6(t) F7(t) M76(t) Frame Match F7(t) to F6(t) M76(t) isan estimate of F6(t) from F7(t) F4(t) FSUM67(t) MSUM67(t) Frame MatchFSUM67(t) to F4(t) Normalize FSUM67(t) with divide by 2 prior to FrameMatch. MSUM67(t) is an estimate of F4(t) from both F6(t) and F7(t) F4(t)FSUM0123(t) MSUM0123(t) Frame Match FSUM0123(t) to F4(t) NormalizeFSUM(t − 4) with divide by 4 prior to Frame Match. MSUM0123(t) is anestimate of F4(t) from F0(t), F1(t), F2(t), F3(t) (or equivalently, F4(t− 4), F5(t − 4), F6(t − 4), F7(t − 4))

In general, one approach taken by the frame match module 402 is toperform block matching on blocks of pixels (e.g., 8×8) in the luminancechannel, and to export the motion vectors from the block matching of theluminance channel for re-use in the chrominance channels. In oneembodiment, the reference image LREF is partitioned into a set of 8×8non-overlapping blocks. The final result of frame-matching is a set ofmotion vectors into the non-reference frame, LF, for each 8×8 block ofLREF to be matched. Each motion vector represents the 8×8 block ofpixels in LF which most closely match a given 8×8 block of LREF. Theluminance pixels are filtered with a binomial filter and decimated priorto block matching.

The luma ME 502 comprises logic to provide the filtering out of noise,coarse block matching (using a multi-level or multi-stage hierarchicalapproach that reduces computational complexity) of the filtered blocks,and refined block matching using undecimated pixel blocks and a finalmotion vector derived from candidates of the coarse block matchingprocess and applied to unfiltered pixels of the inputted frames.Explaining in further detail and with reference to FIG. 5, the luminanceinput (LREF, LF) is received at binomial filter 510, luma refinement BMmodule 518, and luma MC module 522. The binomial filter 510 processesthe data and produces full-pixel luminance (BF_LF, BF_LREF), each of theluminance images of size N_(ver)×N_(hor). The binomial filter 510performs a 2D convolution of each input frame according to the followingequation:

$\begin{matrix}{{{{BF\_ X}\; \left( {i,j} \right)} = {\sum\limits_{m = {- 1}}^{1}{\sum\limits_{n = {- 1}}^{1}{{x\left( {m,n} \right)}{G\left( {{i - m},{j - n}} \right)}}}}},} & {{Eq}.\mspace{14mu} (2)}\end{matrix}$

where x(0: N_(ver)−1, 0: N_(hor)−1) is an input image of sizeN_(ver)×N_(hor), BF_X(i,j) is the binomial filtered output image, andG(m,n) is the 2D convolution kernel given by the following equation:

$\begin{matrix}{{G\left( {i,j} \right)} = {\frac{1}{16}\begin{pmatrix}1 & 2 & 1 \\2 & 4 & 2 \\1 & 2 & 1\end{pmatrix}}} & {{Eq}.\mspace{14mu} (3)}\end{matrix}$

Accordingly, LF and LREF are both binomial filtered according to Eq. (3)to produce BF_LF and BF_LREF, respectively, which are also input to thedecimator 512 and the full BM module 516. Although a binomial filter isdescribed as one example pixel filter, in some embodiments, other typesof filters may be employed without undue experimentation, as should beunderstood by one having ordinary skill in the art.

BF_LF and BF_LREF are received at decimator 512, which performs, in oneembodiment, a decimation by two (2) function in both the vertical andhorizontal dimensions to produce BF_LF2 and BF_LREF2, respectively. Theoutput of the decimator 512 comprises filtered, decimated luminance data(BF_LF2, BF_LREF2), where each of the luminance images are of sizeN_(ver)/2×N_(hor)/2. Thus, if the size of LF and LREF are bothN_(ver)×N_(hor) pixels, then the size of BF_LF2 and BF_LREF2 areN_(ver)/2×N_(hor)/2 pixels. In some embodiments, other factors orfunctions of decimation may be used, or none at all in some embodiments.

The decimated, binomial-filtered luma pixels, BF_LF2 and BF_LREF2, areinput to the DECBM module 514, which performs decimated block matchingon the filtered, decimated data (BF_LF2, BF_LREF2). In one embodiment,the DECBM module 514 applies 4×4 block matching to the 4×4 blocks ofBF_LREF2 to correspond to the 8×8 blocks of BF_LREF. In other words, the4×4 pixel blocks in the decimated domain correspond to 8×8 blocks in theundecimated domain. The DECBM module 514 partitions BF_LREF2 into a setof 4×4 blocks given by the following equation:

$\begin{matrix}{{{{BREF}\; 2\left( {i,j} \right)} = {{BF\_ LREF}\; 2\left( {{4\; {i:{{4\; i} + 3}}},{4\; {j:{{4\; j} + 3}}}} \right)}},{{{where}\mspace{14mu} i} = 0},1,{{\ldots \mspace{14mu} \frac{N_{hor} - 1}{4}\mspace{14mu} {and}\mspace{14mu} j} = 0},1,{\ldots \mspace{14mu} {\frac{N_{ver} - 1}{4}.}}} & {{Eq}.\mspace{14mu} (4)}\end{matrix}$

It is assumed that BF_LREF2 is divisible by four (4) in both thevertical and horizontal dimensions. The set of 4×4 blocks BREF2(i,j) inEq. (4) includes all pixels of BF_LREF2 partitioned as non-overlapping4×4 blocks. A function of the DECBM module 514 is to match each of theseblocks to the most similar blocks in BF_LF2.

For each of the 4×4 blocks at BREF2(i,j), the DECBM module 514 searches,in one example embodiment, over a ±24 horizontal by ±12 vertical searcharea of BF_LF2 (for a total 49×25 decimated pixel area) to find 4×4pixel blocks which most closely match the current 4×4 block. In someembodiments, differently configured (e.g., other than 24×12) searchareas are contemplated. The search area of BF_LF2 is co-located with theblock BREF2(i,j) to be matched. In other words, the search regionBF_LF2_SEARCH(i, j) may be defined by the following equation:

BF _(—) LF2_SEARCH(i,j)=BF _(—) LF2(4i:4i±12,4j:4j±24)  Eq. (5)

Eq. (5) defines the search region as a function of (i j) that iscentered at the co-located block BF_LF2(4i,4j) as in BF_LREF2(4i,4j), orequivalently BREF2(i,j). The search region may be truncated for blocksnear the borders of the frame where a negative or positive offset doesnot exist. Any 4×4 block at any pixel position in BF_LF2_SEARCH(i,j) isa candidate match. Therefore, the entire search area is traversedextracting 4×4 blocks, testing the match, then moving one (1) pixelhorizontally or one (1) pixel vertically. This operation is well-knownto those with ordinary skill in the art as “full search” block matching,or “full search” motion estimation.

One matching criterion, among others in some embodiments, is defined asa 4×4 Sum-Absolute Difference (SAD) between the candidate block inBF_LF2 search area and the current BREF2 block to be matched accordingto Eq. 6 below:

${S\; A\; D\; 4x\; 4\left( {y,x} \right)} = {\sum\limits_{u = 0}^{3}{\sum\limits_{v = 0}^{3}{{{{BF\_ LF}\; 2\left( {{{4\; i} + y + u},{{4\; j} + x + v}} \right)} - {{BREF}\; 2\left( {{i + u},{j + v}} \right)}}}}}$

(Eq. (6)), where −24≦x≦24, −12≦y≦12. The values of y and x whichminimize the SAD 4×4 function in Eq. (6) define the best matching blockin BF_LF2. The offset in pixels from the current BREF2 block in thevertical (y) and horizontal (x) directions defines a motion vector tothe best matching block. If a motion vector is denoted by my, then mv.ydenotes the motion vector vertical direction, and mv.x denotes thehorizontal direction. Note that throughout the present disclosure,reference is made to SAD and SAD computations for distance or differencemeasures.

It should be understood by one having ordinary skill in the art thatother such difference measures, such as sum-squared error (SSE), amongothers well-known to those having ordinary skill in the art can be usedin some embodiments, and hence the example embodiments described hereinand/or otherwise contemplated to be within the scope of the disclosureare not limited to SAD-based difference measures.

In one embodiment, the DECBM module 514 may store not only the bestmotion vector, but a set of candidate motion vectors up toN_BEST_DECBM_MATCHES, where N_BEST_DECBM_MATCHES is a parameter havingan integer value greater than or equal to one. For example, in oneimplementation, N_BEST_DECBM_MATCHES=3. As the DECBM module 514traverses the search area, computing SAD 4×4 according to Eq. (6), theDECBM module 514 keeps track of the N_BEST_DECBM_MATCHES (e.g., minimumSAD 4×4 blocks) by storing the motion vectors (x and y values)associated with those blocks and the SAD 4×4 value. In one embodiment, ablock is only included in the N_BEST_DECBM_MATCHES if its distance ineither of the horizontal or vertical directions is greater than one (1)from any other saved motion vectors. The output of the DECBM module 514is a set of motion vectors (MV_BEST) corresponding toN_BEST_DECBM_MATCHES, the motion vectors input to the full pixel BMmodule 516. In some embodiments, in addition to MV_BEST, the DECBMmodule 514 adds one or more of the following two motion vectorcandidates, if they are not already in the MV_BEST set: a zero motionvector and/or a motion vector of a neighboring block (e.g., a blocklocated one row above). For example, if N_BEST_DECBM_MATCHES=3, meaningthree (3) candidate motion vectors come from the DECBM module 514, thenthe total candidate motion vectors is five (5) (three (3) from the DECBMmodule SAD 4×4 operation plus a zero motion vector plus a neighboringmotion vector). Therefore, in this example, if N_BEST_DECBM_MATCHES=3,then the total candidate motion vectors is five (5).

The DECBM module 514 omits from a search the zero vector (mv.x=0,mv.y=0) as a candidate motion vector output and any motion vectorswithin one pixel of the zero vector since, in one embodiment, the zerovector is always input to the next stage processing in addition to thecandidate motion vectors from the DECBM module 514. Therefore, ifN_BEST_DECBM_MATCHES=3, then the three (3) motion vectors consist ofnon-zero motion vectors, since any zero motion vectors are omitted fromthe search. Any zero-motion vector is included as one of the addedmotion vectors of the output of the DECBM module 514. In someembodiments, zero vectors are not input to the next stage and/or are notomitted from the search. The full pixel BM module 516 receives the setof candidate motion vectors, MV_BEST, and performs a limited, full pixelblock matching using the filtered, undecimated frames (BF_LF, BF_LREF).In other words, the full pixel BM module 516 takes as input the motionvectors obtained in the DECBM module-implemented process, in addition tothe zero motion vector and the motion vector from a neighboring block asexplained above, and chooses a single refined motion vector from thecandidate set. In some embodiments, a neighboring motion vector is notincluded as a candidate.

Operation of an embodiment of the full pixel BM module 516 is describedas follows. The full pixel BM module 516 partitions BF_LREF into 8×8blocks corresponding to the 4×4 blocks in BF_REF2 as follows:

$\begin{matrix}{{{{{BREF}\left( {i,j} \right)} = {{BF\_ LREF}\left( {{8\; {i:{{8\; i} + 7}}},{8\; {j:{{8\; j} + 7}}}} \right)}},{where}}{{i = 0},1,{\ldots \mspace{14mu} \frac{N_{hor} - 1}{8}},\mspace{14mu} {{{and}\mspace{14mu} j} = 0},1,{\ldots \mspace{14mu} {\frac{N_{ver} - 1}{8}.}}}} & {{Eq}.\mspace{14mu} (7)}\end{matrix}$

BREF is a set of non-overlapping 8×8 blocks comprising the entireluminance frame BF_LREF with a direct correspondence to the BREF2 4×4blocks. The full pixel BM module 516 receives the MV_BEST motion vectorsfrom the DECBM module 514 and the full pixel (undecimated), binomialfiltered luminance BF_LREF and BF_LF.

The input, MV_BEST, to the full pixel BM module 516, may be denotedaccording to the following set: MV_BEST={mv_best(0), mv_best(1), . . .mv_best(N_BEST_DECBM_MATCHES−1)}. The full pixel BM module 516 scalesthe input motion vectors to full pixel by multiplying the x and ycoordinates by two (2), according to Eq. (8) as follows:

mvfull(k).x=2×mv_best(k).x 0≦k≦N_BEST_DECBM_MATCHES,

mvfull(k).y=2×mv_best(k).y 0≦k≦N_BEST_DECBM_MATCHES  (Eq. (8))

where 0≦k≦N_BEST_DECBM_MATCHES−1. Note that the zero motion vector andneighboring block motion vector do not need scaling since zero does notneed scaling, and the neighboring block motion vector is already scaled(sourced from the full pixel BM module 516). After scaling to fullpixel, the full pixel BM module 516 determines a refined motion vector,mvrfull(k), from its corresponding candidate motion vector, mvfull(k),by computing a minimum SAD for 8×8 full-pixel blocks in a 5×5 refinementsearch around the scaled motion vector according to Eq. (9):

$\begin{matrix}{{S\; A\; D\; 8\; x\; 8\left( {i,j,{{{{mvfull}(k)} \cdot y} + m},{{{{mvfull}(k)} \cdot x} + n}} \right)} = {\sum\limits_{u = 0}^{7}{\sum\limits_{v = 0}^{7}{{{{BF\_ LF}\left( {{{8\; i} + {{{mvfull}(k)} \cdot y} + n + u},{{8\; j} + {{{mvfull}(k)} \cdot x} + m + v}} \right)} - {{BREF}\left( {{i + u},{j + v}} \right)}}}}}} & {{Eq}.\mspace{14mu} (9)}\end{matrix}$

where −2≦m≦2, −2≦n≦2. Note that one having ordinary skill in the artshould understand in the context of the present disclosure thatrefinement search ranges other than 5×5 are possible and hencecontemplated for some embodiments. Minimizing Eq. (9) for each motionvector of the candidate set results in (N_BEST_DECBM_MATCHES+2) refinedcandidate motion vectors, mvrfull(k), where0≦k<(N_BEST_DECBM_MATCHES+2). The full pixel BM module 516 selects afinal winning motion vector from the refined motion vectors by comparingthe SAD of the refined motion vectors according to the followingequation:

$\begin{matrix}{\lbrack{kf}\rbrack = {\underset{k}{Min}\left\{ {{\lambda*{{MVDIST}(k)}} + {{DISTBIAS}(k)} + {S\; A\; D\; 8x\; 8\left( {i,j,{{{mvrfull}(k)} \cdot y},{{{mvrfull}(k)} \cdot x}} \right)}} \right\}}} & {{Eq}.\mspace{14mu} (10)}\end{matrix}$

where:kf=index of refined motion vector that is the winner;MVDIST(k)=min(dist(mvrfull(k), 0), dist(mvrfull(k), mvfull(B)));mvrfull(B)=winning motion vector of neighboring block;dist(a,b)=distance between motion vectors a and b;min(x,y)=minimum of x and y;DISTBIAS(k)=0 for MVDIST(k)<12, 20 for MVDIST(k)<20, 40 otherwiseλ=operational parameter, e.g. λ=4

In other words, the larger the motion vector, the lower the SAD value tojustify the larger motion vector as a winning candidate. For instance,winners comprising only marginally lower SAD values likely results inrandom motion vectors. The 12, 20, 40 values described above forces anincreased justification (lower SAD values) for increasingly largermotion vectors. In some embodiments, other values and/or other relativedifferences between these values may be used (e.g., 1, 2, 3 or 0, 10,20, etc.). Therefore, the final motion vector result from the full pixelblock mode operation of full pixel BM module 516 is given by:

mvf(i,j).x=mvrfull(kf).x

mvf(i,j).y=mvrfull(kf).y  Eq. (11)

If the SAD value in Eq. (9) corresponding to the best motion vector ofEq. (11) is above a threshold, T_SAD, the block is flagged as a badblock (e.g., BAD_MC_BLOCK) so that instead of copying the blockindicated by the motion vector in the search frame, the motioncompensation process (described below) copies the original block of thereference frame instead.

The resultant output of the full pixel BM module 516 comprises a singlefinal motion vector, MVF, for each non-overlapping block, which is inputto the luma refinement BM module 518. The luma refinement BM module 518(like the full pixel BM module 516) uses 8×8 block matching since itreceives as input full (non-decimated) images. That is, the lumarefinement BM module 518 refines the motion vectors using originalunfiltered frame data (LREF, LF), or more specifically, takes as inputthe set of motion vectors MVF obtained in the full pixel BM module 516and refines the motion vectors using original unfiltered pixels.Explaining further, the luma refinement BM module 518 partitions theoriginal noisy LREF into 8×8 non-overlapping blocks corresponding to the8×8 blocks in BF_REF according to the following equation:

$\begin{matrix}{{{{{REF}\left( {i,j} \right)} = {{LREF}\left( {{8\; {i:{{8\; i} + 7}}},{8\; {j:{{8\; j} + 7}}}} \right)}},{where}}{{i = 0},1,{\ldots \mspace{14mu} \frac{N_{hor} - 1}{8}},\mspace{14mu} {{{and}\mspace{14mu} j} = 0},1,{\ldots \mspace{14mu} {\frac{N_{ver} - 1}{8}.}}}} & {{Eq}.\mspace{14mu} (12)}\end{matrix}$

REF is a set of non-overlapping 8×8 blocks comprising the entireluminance frame LREF with a direct correspondence to the BREF 8×8blocks. For each block to be matched in REF, there is a motion vectorfrom full pixel block mode operation of full pixel BM module 516 (e.g.,mvf(i,j)). In one embodiment, a 1-pixel refinement around mvf(i,j)proceeds by the m and n which minimizes the following equation:

$\begin{matrix}{{S\; A\; D\; 8\; x\; 8\left( {i,j,{{{{mvf}\left( {i,j} \right)} \cdot y} + m},{{{{mvf}\left( {i,j} \right)} \cdot x} + n}} \right)} = {\sum\limits_{u = 0}^{7}{\sum\limits_{v = 0}^{7}{{{{LF}\left( {{{8\; i} + {{{mvf}\left( {i,j} \right)} \cdot y} + u + n},{{8\; j} + {{{mvf}\left( {i,j} \right)} \cdot x} + v + m}} \right)} - {{REF}\left( {{i + u},{j + v}} \right)}}}}}} & {{Eq}.\mspace{14mu} (13)}\end{matrix}$

where −1≦m≦−1, −1≦n≦−1. In some embodiments, pixel refinement other thanby one (1-pixel) may be used in some embodiments, or omitted in someembodiments. The refined motion vector for the block at position i,j isgiven by the values of m and n, mref and nref respectively, whichminimize Eq. (13). The refined motion vector is given by Eq. (14) asfollows:

$\begin{matrix}{{{{{mvr}\left( {i,j} \right)} \cdot x} = {{{{mvf}\left( {i,j} \right)} \cdot x} + {mref}}}{{{{mvr}\left( {i,j} \right)} \cdot x} = {{{{mvf}\left( {i,j} \right)} \cdot x} + {mref}}}{{{{{mvr}\left( {i,j} \right)} \cdot y} = {{{{{mvf}\left( {i,j} \right)} \cdot x} + {{nref}{{{mvr}\left( {i,j} \right)} \cdot y}}} = {{{{{mvf}\left( {i,j} \right)} \cdot x} + {{nref}{where}i}} = 0}}},1,{\ldots \mspace{14mu} \frac{N_{hor} - 1}{8}},\mspace{14mu} {{{and}\mspace{14mu} j} = 0},1,{\ldots \mspace{14mu} {\frac{N_{ver} - 1}{8}.}}}} & {{Eq}.\mspace{14mu} (14)}\end{matrix}$

MVRL (also referred to herein as refined motion vector(s)) denotes thecomplete set of refined motion vectors mvr(i,j) (e.g., for i=0, 8, 16, .. . N_(ver)−7; j=0, 8, 16, . . . N_(Hor)−7) for the luminance channeloutput from the luma refinement BM module 518. In other words, MVRLdenotes the set of motion vectors representing every non-overlapping 8×8block of the entire frame.

MVRL is used by the luma MC module 522 and the chroma refinement BMmodule 520. Referring to the chroma refinement BM module 520, therefined motion vectors, MVRL, are received at the chroma refinement BMmodule 520, which performs a refined block matching in the chrominancechannel based on inputs CF and CREF. For 4:2:0 video formats, when thechroma is sub-sampled by a factor of two (2) in each of the horizontaland vertical dimensions, the chroma refinement BM module 520 performs4×4 full-pixel block matching. That is, the chroma (both Cb and Cr) 4×4blocks correspond directly to the luma 8×8 blocks. Using the MVRL input,the chroma refinement BM module 520, for both Cb and Cr chroma frames,performs a 1-pixel refinement around the MVRL input motion vectors in asimilar manner to the process performed by the luma refinement BM module518 described above, but using 4×4 blocks instead of 8×8, and a SAD 4×4instead of SAD 8×8 matching criterion. The resulting set of motionvectors are MVRCb for Cb and MVRCr for Cr (collectively shown as MVRC inFIG. 5), which are input to the chroma MC module 524 to perform motioncompensation.

Motion compensation is a well-known method in video processing toproduce an estimate of one frame from another (e.g., to “match” oneframe to another). MVRL and MVRC are input to motion compensation (MC)processes performed at luma MC module 522 and chroma MC module 524,respectively, which import the blocks indicated by the MVRL and MVRCmotion vectors. With reference to the luma MC module 522, after theblock matching has been accomplished as described hereinabove, the LFframe is frame-matched to LREF by copying the blocks in LF indicated bythe motion vectors MVRL. For each block, if the block has been flaggedas a BAD_MC_BLOCK by the full pixel BM module 516, instead of copying ablock from the LF frame, the reference block in LREF is copied instead.For chroma operations, the same process is carried out in chroma MCmodule 508 on 4×4 blocks using MVRCb for Cb and MVRCr for Cr, and hencediscussion of the same is omitted for brevity. Note that 8×8 and 4×4were described above for the various block sizes, yet one havingordinary skill in the art should understand that in some embodiments,other block sizes than those specified above may be used.

Having described the overall operation of an embodiment of the framematch module 402, attention is now directed to the SMC sub-system 300A.In the block matching process, the 8×8 blocks (4×4 for chroma) are inthe same pixel positions in the matched frames M0123, M5, and M67.Accordingly, the “seams” of the 8×8 blocks of the matched frames are inthe same pixel positions. In circumstances where there exists a highlevel of filtering, some of these seams may become visible in thedenoised frames. Further, every pixel is matched with a group of pixelsin its designated block, so there is a bias for that pixel to be groupedwith certain pixels. Even without the seams becoming visible, anundesirable blocking effect may become visible.

Certain embodiments of an SMC sub-system 300A implement a mechanismreferred to herein as staggered motion compensation (SMC), which enables8×8 blocks to be positioned in different positions in the M0123, M5 andM67 frames, and further, in the fields of these frames. As shown inexample frame match module 402, SMC sub-system 300A comprises border addlogic 526 and 528 at the input of the luma ME 502 and chroma ME 504,respectively, and border extract logic 530 and 532 at the output of theluma MC 506 and chroma MC 508, respectively. In general, the border addlogic 526 receives the luma LF and LREF and outputs the correspondingframes to be matched with the added pixel borders to the binomial filter510 and the luma refinement BM 518. The border add logic 528 receivesthe chroma CF and CREF and outputs the corresponding frames with theadded pixels to the chroma refinement BM 520 and the chroma MC 524. Theborder extract logic 530 residing in one embodiment in the luma MC 506receives the output from the luma MC 522 and strips away the added pixelborder and outputs LMAT (without the border). Similarly, the borderextract logic 532 of the chroma MC 508 receives the output from thechroma MC 524, strips away the added pixel border, and outputs CMAT(without the border). Each of the above-described logic (e.g., 526, 528,530, and 532) of the SMC sub-system 300A receives border configuration(BC) signals from a processor (not shown) in or associated with the VDNsystem.

Explaining further, to accomplish SMC in one embodiment, a border isplaced by the border add logic 526 and 528 around the frames to bematched (LF and LREF, CF and CREF in FIG. 5). The total amount of leftborder pixels plus the total amount of right border equals, in thisexample, eight (8) for luma and four (4) for chroma for any of thematching frames (and same for top/bottom border). For example, a borderof 2-left, 6-right, 4-top, 4-bottom is one example luma borderconfiguration signaled (e.g., by the BC or border_configuration) to theborder add logic 526 of the SMC sub-system 300A. Note that the BC isdownsampled by two (2) in the border add logic 528 so that the chromaeffectively has the same BC as the luma. The border itself in oneembodiment is simply black pixels. One having ordinary skill in the artshould understand, in the context of the present disclosure, that otherquantities of pixels may be used for the border.

For interlaced video, each field is block matched separately. Therefore,a tally of two (2) different border configurations per frame iscontemplated in some embodiments, one for each field. For example, sinceM0123, M5 and M67 are the result of three (3) frame-matching operations,there may exist a total of six (6) border configurations (e.g.,signalled to the SMC logic), putting the seams in six (6) differentlocations. Some example border configurations for luma are given inexample Table 600 shown in FIG. 6A, which shows example pixel borderconfigurations for the top, bottom, left, and right of each field frameto be matched. It should be appreciated, in the context of the presentdisclosure, that for these or other quantities of frame matchingoperations, a different quantity of border pixel configurations may beapplied.

An example constellation 650 of the six (6) border configurations isshown in FIG. 6B (denoted symbolically by “X”). As shown, the borderconfigurations of Table 600 evenly spread out the seams of the blockmatching process. It is further noted that the border configurations ofTable 600 are all even amounts, since in at least one embodiment, thechroma block matching uses exactly the same border downsampled by two(2) to properly re-use motion vectors. Therefore, for chroma, the borderconfigurations are the same as Table 600 divided by two (2). As anexample, M5 for chroma on Field 1 (F1) uses [2,2,2,2]. Also note thatthe border size for chroma is four (4) pixels total for left-right, andfour (4) pixels total for top/bottom. For progressive video, there isonly one (1) large field, so only the F1 border configuration is used.

After block matching, the border is stripped away (e.g., removed) by theborder extract logic 530 and 532 and has no effect on the rest of theoverlapped block processing (e.g., is transparent to the overlappedblock processing module 350).

With reference to FIG. 3, having described an example embodiment of thevarious modules or logic that comprise the frame alignment module 310for the VDN system embodiment 200 c−1, attention is directed to theoverlapped block processing module 350. In general, after framematching, the overlapped block processing module 350 denoises theoverlapped 3D blocks and accumulates the results, as explained inassociation with FIGS. 2A-2C. In one embodiment, looping occurs bystepping j by a step size s in pixels (e.g. s=2). The overlapped blockprocessing moves horizontally until j==N_(Hor)−1, after which j is setto 0 and i is incremented by s. Note that in some embodiments, seven (7)pixels are added around the border of the frame to enable the borderpixels to include all blocks. For simplicity, these pixels may be a DCvalue equal to the touching border pixel.

Before proceeding further on the discussion of the overlapped blockprocessing loop, some VDN system embodiments include preprocessingsub-systems 300 that utilize single channel field processing asindicated above. Accordingly, in one embodiment, it is noted that anoutput of the frame alignment module 310 is provided to the field-frameprocessing (FFP) sub-system 300C. In one embodiment, for field-frameprocessing, the field-frame mapping logic 380 partitions (e.g.,logically partitions) a matched frame into plural superblocks, eachsuperblock comprising in one embodiment 8×16 non-overlapping pixelblocks (e.g., 8-pixel horizontal, 16-pixel vertical). An 8×16non-overlapping pixel block comprises two (2) sub-blocks (e.g., each8×8, collectively spanning sixteen (16) pixels vertically for luma). Forchroma, both dimensions are halved, though in some embodiments, otherinteger decimations are contemplated for chroma processing. Thefield-frame mapping logic 380 of the FFP sub-system 300C provides (e.g.,computes) a field-frame map (e.g., once) for the entire F4 frame with abinary indication 1=field, 0=frame for each of the non-overlapping 8×16superblocks from the luma as explained below. The chroma processing usesthe luma field-frame map, though in some embodiments, a separatefield-frame map for chroma may be provided.

Let I(x,y) denote the luminance intensity for a pixel at position x, yfor all non-overlapping superblocks at positions x=0, 16, 32, 48, . . .and y=0, 8, 16, 24, 32 . . . and compute A_(frame), A_(field) asfollows:

$\begin{matrix}{A_{frame} = {\sum\limits_{x = 0}^{13}{\sum\limits_{y = 0}^{7}{{{I\left( {x,y} \right)} - {I\left( {{x + 1},y} \right)}}}}}} & (a) \\{A_{field} = {\sum\limits_{x = 0}^{13}{\sum\limits_{y = 0}^{7}{{{I\left( {x,y} \right)} - {I\left( {{x + 2},y} \right)}}}}}} & (b)\end{matrix}$

Further, consider, A_(diff) given by:

A _(diff) =A _(frame) −A _(field)  (c)

If A_(diff)>TADIFF, then the 8×16 luma superblock is marked as a fieldblock, otherwise the 8×16 luma superblock is marked as a frame block,where TADIFF is an operational parameter that may be set once at startup(e.g. TADIFF=50), or in some embodiments set more often and/or atdifferent stages. One having ordinary skill in the art shouldunderstand, in the context of the present disclosure, that thoughequations (a) through (c) provide one method for determining a field orframe, other mechanisms known to those having ordinary skill in the artmay be employed and hence are contemplated to be within the scope of thedisclosure.

The field-frame map is used in the overlapped block processing (e.g.,AVC-DCT, Haar, thresholding, Inverse, etc.) as explained in furtherdetail below. In general, for each overlapped block, a field-frameindication is derived from the field-frame map. If the indication isfield, then the two vertically interleaved 8×8 field blocks of thesuperblock are simultaneously processed by the overlapped blockprocessing for luma. In other words, there is a direct coupling of theprocessing of fields, as opposed to splitting and processing as separatefields. If the indication is frame, then two 8×8 frame blocks (e.g.,top/bottom) are processed for luma. As indicated above, for oneembodiment, chroma processing uses the field-frame map generated for theF4 frame from luma and performs analogous processing on, in oneembodiment, 4×4 fields for frame blocks. Note that at a timecorresponding to completion of (e.g., after) the overlapped blockprocessing, the denoised F4 blocks are accumulated in the appropriateposition in the accumulation buffer (e.g., either as two verticallyinterleaved sub-blocks of the superblock or top/bottom adjacentsub-blocks of the superblock).

Explaining the above in further detail, it is noted that onecomplication in the transition from field-frame processing to overlappedblock processing is that the field-frame map corresponds to field-frameindications for non-overlapping 8×16 superblocks, and as explainedfurther below, overlapped block processing contains many more blocks dueto the overlapping. As shown in FIG. 7, an overlapped block 702 (e.g., asuperblock) intersects four (4) non-overlapping superblocks in thefield-frame map 700. One approach implemented by the field-frame logic380 of FIG. 3 comprises the following procedure: for each overlappedblock 702, if the overlapped block position in the frame intersects anyfield blocks 704 (e.g., field-designated superblocks, where the fielddesignation is represented by “FI” in FIG. 7) in the field-frame 700,the overlapped block 702 (e.g., the sub-blocks of the overlapped blockor superblock 702) is marked for field processing. If the overlappedblock position overlaps only frame blocks 706 (e.g., frame-designatedsuperblocks, where the frame designation is represented by “FR” in FIG.7) in the field-frame map 700, then the block 702 (e.g., the sub-blocksof the overlapped block or superblock 702) is marked for frameprocessing. For progressive video input, the field-frame map 700 is setto all ones (1 s) indicating frame processing only.

Further, note that for a step size vertically (same as horizontally) oftwo (2) in the overlapped block processing, there are 4×4 or sixteen(16) overlapped blocks accumulated for all blocks except the borders. Onthe bottom border, both the bottom row of 8×8 blocks and the rowimmediately above have no vertical step, so these blocks (other thanleft/right borders) have four (4) blocks accumulated instead of sixteen(16). This situation arises because the overlapped processing works on8×16 blocks, and it is not possible to shift vertically once theprocessing has reached the bottom sixteen (16) pixels of the frame.

Block matching (motion estimation and compensation) describedhereinbefore is unaffected by field-frame processing, since all blockmatching proceeds on two independent fields (i.e., there is nofield-frame in block matching). For progressive inputs, block matchingproceeds using one large field per frame instead of two (2) fields(i.e., there is no field processing).

Referring again to FIG. 3, and assuming hereinafter frame processing(since if field processing is assumed, the only significant effect isthat there should be a communication to the 2D buffer accumulation as tothe presence of interleaved field blocks or top/bottom frame blocks toenable a determination of where to accumulate the blocks), theoverlapped block processing module 350 receives the matched framesM4(t), M5(t), MSUM67(t), and MSUM0123(t), and extracts co-located blocksof 8×8 pixels from these four (4) frames. Denote the four (4) blocksextracted at a particular i, j pixel position in the four (4) frames asb(i, j, t) with t=0 . . . 3, then after reordering of the blocks (asexplained below in the context of the 1D transform illustrated in FIGS.8A-8D), the following terminology is described:

b(i, j, 0) is an 8×8 block from MSUM0123(t) at pixel position i, jb(i, j, 1) is an 8×8 block from M4(t) at pixel position i, jb(i, j, 2) is an 8×8 block from M5(t) at pixel position i, jb(i, j, 3) is an 8×8 block from MSUM67(t) at pixel position i, j

Starting with i=0 and j=0, the upper left corner of the frames, a 2Dtransform module 304 (also referred to herein as transform logic)extracts the four (4) blocks b(0, 0, 0:3) and performs a 2D transform.That is, the 2D transform is taken on each of the four (4) temporalblocks b(i, j, t) with 0≦t≦3, where i, j is the pixel position of thetop left corner of the 8×8 blocks in the overlapped block processing. Insome embodiments, a 2D-DCT, DWT, among other well-known transforms maybe used as the spatial transform. In an embodiment described below, the2D transform is based on an integer DCT defined in the Advanced VideoCoding (AVC) standard (e.g., an 8×8 AVC-DCT), which has the followingform:

$\begin{matrix}{{{H(X)} = {{{DCT}(X)} = {C \cdot X \cdot C^{T}}}}{{where},{C = {\begin{bmatrix}8 & 8 & 8 & 8 & 8 & 8 & 8 & 8 \\12 & 10 & 6 & 3 & {- 3} & {- 6} & {- 10} & {- 12} \\8 & 4 & {- 4} & {- 8} & {- 8} & {- 4} & 4 & 8 \\10 & {- 3} & {- 12} & {- 6} & 6 & 12 & 3 & {- 10} \\8 & {- 8} & {- 8} & 8 & 8 & {- 8} & {- 8} & 8 \\6 & {- 12} & 3 & 10 & {- 10} & {- 3} & 12 & {- 6} \\4 & {- 8} & 8 & {- 4} & {- 4} & 8 & {- 8} & 4 \\3 & {- 6} & 10 & {- 12} & 12 & {- 10} & 6 & {- 3}\end{bmatrix} \cdot {1/8}}}}} & {{Eq}.\mspace{14mu} (15)}\end{matrix}$

X is an 8×8 pixel block, and the products on the right are matrixmultiplies. One method for computing the DCT employs the use of signedpowers of two for computing the multiplication products. In this way nohardware multipliers are needed; but rather, the products are created byshifts and additions thereby reducing the overall logic. In someembodiments, hardware multipliers may be used. In addition, scalingfactors may be used, which may also reduce the required logic. Toretrieve the original pixels (X), the integer matrix is scaled such thatthe inverse DCT implemented by the inverse 2D transform module 314yields the original values of X. One form of the inverse AVC-DCT of theinverse transform module 314 comprises the following:

X=(C ^(T) ·H(X)·C)

S _(i,j),  Eq. (16)

or by substitution:

X=C _(S) ^(T) ·H(X)·C _(s),  Eq. (17)

where C_(s)=C

S_(i,j) and the symbol

denotes element by element multiplication. One example of scalingfactors that may be used is given below:

$\begin{matrix}{S_{i,j} = {\frac{1}{\sum\limits_{i,{j = 0}}^{7}C_{({i,j})}^{2}} = \begin{matrix}0.0020 \\0.0017 \\0.0031 \\0.0017 \\0.0020 \\0.0017 \\0.0031 \\0.0017\end{matrix}}} & {{Eq}.\mspace{14mu} (18)}\end{matrix}$

After the 2D transform, the overlapped block processing module 350changes the spatial dimension from 2D to 1D by zig-zag scanning theoutput of each 2D transform from low frequency to highest frequency, sothe 2D transformed block bs(i, j, f) becomes instead bs(zz_index, f),where 0≦zz_index≦63, 0≦f≦3. The mapping of (i, j) to zz_index is givenby the zig_zag_scan vector below, identical to the scan used in MPEG-2video encoding. In some embodiments, the 2D dimension may be retainedfor further processing. If the first row of the 2D matrix is given byelements 0 through 7, the second row by 8 through 16, thenzig_zag_scan[0:63] specifies a 2D to 1D mapping as follows:

zig_zag_scan[0:63] = {  0,1,8,16,9,2,3,10,17,24,32,25,18,11,4,5, 12,19,26,33,40,48,41,34,27,20,13,6,7,14,21,28, 35,42,49,56,57,50,43,36,29,22,15,23,30,37,44,51, 58,59,52,45,38,31,39,46,53,60,61,54,47,55,62,63 };

At a time corresponding to computation of the 2D transform (e.g.,subsequent to the computation), the temporal mode (TemporalMode) isselected by temporal mode module 302 (also referred to herein astemporal mode logic). When utilizing a Haar 1-D transform, theTemporalMode defines whether 2D or 3D thresholding is enabled, whichHaar subbands are thresholded for 3D thresholding, and which spatialsubbands are thresholded for 2D thresholding. The temporal mode mayeither be SPATIAL_ONLY, FWD4, BAK4, or MODE8, as further describedhereinbelow. The temporal mode is signaled to the 2D threshold module306 and/or the 3D threshold module 310 (herein also collectively orindividually referred to as threshold logic or thresholding logic). Ifthe TemporalMode==SPATIAL_ONLY, then the 2D transformed blockbs(zz_index, 1) is thresholded yielding bst(zz_index, 1). If thetemporal mode is not SPATIAL_ONLY, then bst(zz_index, t) is set tobs(zz_index, f).

Following spatial thresholding by the 2D threshold module 306, thebst(zz_index, t) blocks are 1D transformed at the 1D transform module308 (also referred to herein as transform logic), yieldingbhaar(zz_index, f). The 1D transform module 308 takes in samples fromthe 2D transformed 8×8 blocks that have been remapped by zig-zagscanning the 2D blocks to 1D, so that bs(zz_index, f) represents asample at 0≦zz_index≦63 and 0≦f≦3 so that the complete set of samples isbs(0:63, 0:3). Whereas the 2D transform module 304 operates on spatialblocks of pixels from the matched frames, the 1D transform module 308operates on the temporal samples across the 2D transformed frame blocksat a given spatial index 0≦zz_index≦63. Therefore, there are sixty-four(64) 1D transforms for each set of four (4) 8×8 blocks.

As indicated above, the 1D Transform used for the VDN system embodiment200 c−1 is a modified three-level, 1D Haar transform, though not limitedto a Haar-based transform or three levels. That is, in some embodiments,other 1D transforms using other levels, wavelet-based or otherwise, maybe used, including DCT, WHT, DWT, etc., with one of the goals comprisingconfiguring the samples into filterable frequency bands. Beforeproceeding with processing of the overlapped block processing module350, and in particular, 1D transformation, attention is re-directed toFIGS. 8A-8D, which illustrates various steps in the modification of a 1DHaar transform in the context of the reduction in frame matchingdescribed in association with FIGS. 2A-2C. It should be understood thateach filter in the evolution of the 1D Haar shown in respective FIGS.8A-8D may be a stand-alone filter that can be used in some VDN systemembodiments. A modified Haar wavelet transform is implemented by the 1Dtransform module 308 (and the inverse in inverse 1D transform module312) for the temporal dimension that enables frame collapsing in themanner described above for the different embodiments. In general, a Haarwavelet transform in one dimension transforms a 2-element vectoraccording to the following equation:

$\begin{matrix}{{{\begin{pmatrix}{y(1)} \\{y(2)}\end{pmatrix} = {T \cdot \begin{pmatrix}{x(1)} \\{x(2)}\end{pmatrix}}},{where}}{T = {\frac{1}{\sqrt{2}}{\begin{pmatrix}1 & 1 \\1 & {- 1}\end{pmatrix}.}}}} & {{Eq}.\mspace{14mu} (19)}\end{matrix}$

From Eq (19), it is observed that the Haar transform is a sum-differencetransform. That is, two elements are transformed by taking their sum anddifference, where the term 1/√{square root over (2)} is energypreserving, or normalization. It is standard in wavelet decomposition toperform a so-called “critically sampled full dyadic decomposition.”

A signal flow diagram 800A for the Haar transform, including the forwardand inverse transforms, is shown in FIG. 8A. The signal flow diagrams800A, 800B, 800C, or 800D in FIGS. 8A-8D are illustrative of exampleprocessing (from top-down) of 2D transform samples that may beimplemented collectively by the 1D transform module 308 and the inverse1D transform module 312. The signal flow diagram 800A is divided into aforward transform 802 and an inverse transform 804. The normalizing term1/√{square root over (2)} may be removed if the transform is resealed oninverse (e.g., as shown in FIGS. 8A-8D by factors of four (4) and two(2) with ×2 and ×4, respectively, in the inverse transform section 804).In addition, while FIG. 8A shows the inverse Haar transform 804following directly after the forward transform 802, it should beappreciated in the context of the present disclosure that in view of thedenoising methods described herein, thresholding operations (not shown)may intervene in some embodiments between the forward transform 802 andthe inverse transform 804.

In a dyadic wavelet decomposition, a set of samples are “run” throughthe transformation (e.g. as given by Eq. (19)), and the result issubsampled by a factor of two (2), known in wavelet theory as being“critically sampled.” Using eight (8) samples (e.g., 2D transformed,co-located samples) as an example in FIG. 8A, the samples b0 through b7are transformed in a first stage 806 by Eq. (19) pair-wise on [b0 b1],[b2 b3], . . . [b6 . . . b7], producing the four (4), low frequency,sum-subband samples [L00, L01, L02, L03] and four (4), high frequency,difference-subband samples [H00, H01, H02, H03]. Since half the subbandsare low-frequency, and half are high-frequency, the result is what isreferred to as a dyadic decomposition. An eight (8) sample fulldecomposition continues by taking the four (4), low-frequency subbandsamples [L00, L01, L02, L03] and running them through a second stage 808of the Eq. (19) transformation with critical sampling, producing two (2)lower-frequency subbands [L10, L11] and two (2) higher frequencysubbands [H10, H11]. A third stage 610 on the two (2) lowest frequencysamples completes the full dyadic decomposition, and produces [L20,H20].

The 1D Haar transform in FIG. 8A enables a forward transformation 802and inverse transformation 804 when all samples (e.g., bd(i, j, 0:7))are retained on output. This is the case for a 3D accumulation buffer.However, as discussed hereinabove, the output of the 2D+1 accumulationbuffer (see FIGS. 2A-2B) requires only bd(i, j, 4) and bd(i, j, 7).Therefore, simplification of the flow diagram of 800A, where only thebd(i, j, 4) and bd(i, j, 7) are retained, results in the flow diagramdenoted as 800B and illustrated in FIG. 8B.

In FIG. 8B, since bd(i, j, 4) and bd(i, j, 7) are the desired outcome,the entire left-hand side of the transformation process requires onlythe summation of the first four (4) samples b(i, j, 0:3). As describedbelow, this summation may occur outside the 1D transform (e.g., as partof the frame matching process). On the right side, there has been are-ordering of the samples when compared to the flow diagram of FIG. 8A(i.e., [b4 b5 b6 b7] to [b4 b7 b6 b5]). In addition, only the sum ofb(i, j, 5) and b(i, j, 6) is required (not a subtraction). Accordingly,by using this simplified transform in flow diagram 800B, with samplereordering, and frame-matching the sum of the first four (4) frames tothe reference frame F4(t), the first four (4) frames may be collapsedinto a single frame. For frames 5 and 6, an assumption is made that bothframes have been frame matched to the reference producing M5(t) andM6(t), and hence those frames may be summed prior to (e.g., outside) the1D transform.

When the summing happens outside the 1D transform, the resulting 1Dtransform is further simplified as shown in the flow diagram 800C ofFIG. 8C, where b0+b1+b2+b3 represents the blocks from the frame-matchedsum of frames F0(t) through F3(t), that is MSUM0123(t), prior to the 1Dtransform, and b5+b6 represents the sum of blocks b(i, j, 5) and b(i, j,6) from frame-matched and summed frames M5(t) and M6(t), that isMSUM56(t), the summation implemented prior to the 1D transform. Notethat for 2D+1 accumulation embodiments corresponding to Haarmodifications corresponding to FIGS. 8B and 8C, there is a re-orderingof samples such that b7 is swapped in and b5 and b6 are moved over.

Referring to FIG. 8D, shown is a flow diagram 800D that is furthersimplified based on a 2D accumulation buffer, as shown in FIG. 2C. Thatis, only one input sample, bd4, has been retained on output to the 2Daccumulation buffer. Note that in contrast to the 2D+1 accumulationbuffer embodiments where sample re-ordering is implemented as explainedabove, the Haar modifications corresponding to FIG. 8D involve no samplere-ordering.

Continuing now with the description pertaining to 1D transformation inthe overlapped block processing module 350, and with reference to FIG.3, at each zz_index, the 1D transform 308 processes the four (4) samplesbs(zz_index, 0:3) to produce bhaar(0:63, 0:3), which includes Haarsubbands [L20, H20, H02, H11]. The first index is from the stages 0, 1,2, so L20 and H20 are from the third stage 810 (FIG. 8D), H02 is fromthe first stage 806 (FIG. 8D), and H11 is from the second stage (808).These Haar subbands are as illustrated in FIG. 8D and are furtherinterpreted with respect to the matched frames as follows:

L20: summation of all matched blocks b(i,j, 0:7);

H20: difference between matched blocks in MSUM0123(t) and the total sumof matched blocks in M4(t), M5(t) and 2×MSUM67(t);

H02: For 2D+1 Accumulation Buffer (FIG. 8C), difference between matchedblocks b4 and b7 from frames M4(t), and M7(t), respectively. For 2DAccumulation Buffer (FIG. 8D), difference between matched blocks b4 andb5 from frames M4(t), and M5(t), respectively.

H11: For 2D+1 Accumulation Buffer (FIG. 8C), difference between sum(b4+b7) matched blocks from M4(t) and M7(t) respectively, and2×MSUM56(t). For 2D Accumulation Buffer (FIG. 8D), difference betweensum (b4+b5) matched blocks from M4(t) and M5(t) respectively, and2×MSUM56(t).

With reference to FIG. 9, shown is a schematic diagram 900 thatconceptually illustrates the various temporal mode selections thatdetermine whether 3D or 2D thresholding is enabled, and whether Haar orspatial subbands are thresholded for 3D and 2D, respectively. As shown,the selections include MODE8, BAK4, FWD4, and SPATIAL, explained furtherbelow. The temporal mode selections make it possible for the VDN systems200 to adapt to video scenes that are not temporally correlated, such asscene changes, or other discontinuities (e.g., a viewer blinks his orher eye or turns around during a scene that results in the perception ofa discontinuity, or discontinuities associated with a pan shot, etc.) byenabling a determination of which frames of a given temporal sequence(e.g., F0(t)-F7(t)) can be removed from further transform and/orthreshold processing. The TemporalMode selections are as follows:

For TemporalMode==MODE8 or TemporalMode==SPATIAL: (no change). In otherwords, for TemporalMode set to MODE8 or SPATIAL, there is no need topreprocess the samples before the 1D transform. For FWD4 or BAK4temporal modes, the samples undergo preprocessing as specified below.

For TemporalMode==FWD4: (Input sample b0+b1+b2+b3 from MSUM0123(t) isset equal to zero);

For TemporalMode==BAK4: (Input sample b4 set equal to 4*b4);

In one embodiment, a temporal mode module 302 computes TemporalModeafter the 2D transform is taken on the 8×8×4 set of blocks. TheTemporalMode takes on one of the following four values:

(a) SPATIAL: when the TemporalMode is set to SPATIAL, 2D (spatial)thresholding takes place after the 2D transform on each 2D blockbs(0:63, t) 0≦t≦3 separately to produce bst(0:3, t). In other words,under a spatial temporal mode, there is no 1D transformation orthresholding of 3D blocks (temporal dimension is removed for thisiteration). If the Temporal Mode is not set to SPATIAL, then bst(0:63,t) is set to bs(0:63, t) (pass-through).

(b) FWD4: when the TemporalMode is set to FWD4, right-sided (later)samples from M4(t), M5(t) and MSUM67(t) are effectively used, andsamples from the left (earlier) side, in MSUM0123(t), are not used.

(c) BAK4: when the TemporalMode is set to BAK4, left-sided (earlier)samples from MSUM0123(t) and M4(t) are effectively used, and samplesfrom the right (later) side, in M5(t) and MSUM67(t), are not used.

(d) MODE8: when the TemporalMode is set to MODE8, all samples are used.The TemporalMode is computed for every overlapped set of blocks (e.g.,its value is computed at every overlapped block position). Therefore,implicitly, TemporalMode is a function of i, j, so TemporalMode(i, j)denotes the value of TemporalMode for the i, j-th pixel position in theoverlapped block processing. The shorthand, “TemporalMode” is usedthroughout herein with the understanding that TemporalMode comprises avalue that is computed at every overlapped block position of a givenframe to ensure, among other reasons, proper block matching is achieved.In effect, the selected temporal mode defines the processing (e.g.,thresholding) of a different number of subbands (e.g., L20, H20, etc.).

Having described the various temporal modes implemented in the VDNsystems 200, attention is now directed to a determination of whichtemporal mode to implement. To determine TemporalMode, the SubbandSAD iscomputed (e.g., by the 2D transform module 304 and communicated to thetemporal mode module 302) between the blocks bs(0:63, k) and bs(0:63, 1)for 0≦k≦3 (zero for k=1), which establishes the closeness of the matchof co-located blocks of the inputted samples (from the matched frames),using the low-frequency structure of the blocks where the signal canmostly be expected to exceed noise. Explaining further, thedetermination of closeness of a given match may be obscured or skewedwhen the comparison involves noisy blocks. By rejecting noise, the levelof fidelity of the comparison may be improved. In one embodiment, theVDN system 200 c-1 effectively performs a power compaction (e.g., aforward transform, such as via DCT) of the blocks at issue, whereby mostof the energy of a natural video scene are power compacted into a few,more significant coefficients (whereas noise is generally uniformlydistributed in a scene). Then, a SAD is performed in the DCT domainbetween the significant few coefficients of the blocks under comparison(e.g., in a subband of the DCT, based on a predefined threshold subbandSAD value, not of the entire 8×8 block), resulting in removal of asignificant portion of the noise from the computation and henceproviding a more accurate determination of matching.

Explaining further, in one embodiment, the subbandSAD is computed usingthe ten (10) lowest frequency elements of the 2D transformed blocksbs(0:9, 0:3) where the frequency order low-to-high follows the zig-zagscanning specified hereinbefore. In some embodiments, fewer or greaternumbers of lowest frequency elements may be used. Accordingly, for thisexample embodiment, the SubbandSAD(k) is given by the followingequation:

$\begin{matrix}{{{{SubbandSAD}(k)} = {\sum\limits_{z = 0}^{9}{{{{bs}\left( {z,k} \right)} - {{bs}\left( {z,1} \right)}}}}},} & {{Eq}.\mspace{14mu} (20)}\end{matrix}$

where 0≦k≦3, and SubbandSAD(1)=0.

Integer counts SubbandFWDCount4, SubbandBAKCount4 and SubbandCount8 maybe defined as follows:

$\begin{matrix}{{{SubbandCountFWD}\mspace{14mu} 4} = {\sum\limits_{k = 1}^{3}{{SetToOneOrZero}\left( {{{SubbandSAD}\mspace{11mu} (k)} < {Tsubbandsad}}\; \right)}}} & {{Eq}.\mspace{14mu} \left( {21a} \right)} \\{{{SubbandCountBAK}\mspace{11mu} 4} = {\sum\limits_{k = 0}^{1}{{SetToOneOrZero}\left( {{{SubbandSAD}\mspace{11mu} (k)} < {Tsubbandsad}}\; \right)}}} & {{Eq}.\mspace{14mu} \left( {21b} \right)} \\{{{SubbandCount}\mspace{14mu} 8} = {\sum\limits_{k = 0}^{3}{{SetToOneOrZero}\left( {{{SubbandSAD}\mspace{11mu} (k)} < {Tsubbandsad}}\; \right)}}} & {{Eq}.\mspace{14mu} \left( {21c} \right)}\end{matrix}$

where the function SetToOneOrZero(x) equals 1 when its argumentevaluates to TRUE, and zero otherwise, and Tsubbandsad is a parameter.In effect, Eqns. 21a-21c are computed to determine how many of theblocks are under the subband SAD threshold, and hence determine thetemporal mode to be implemented. For instance, referring to Eq. 21c, forMODE8, the DCT of b0+b1+b2+b3 should be close enough in the lowerfrequencies to the DCT of b4 (and likewise, the DCT of b5 and b6+7should be close enough in the lower frequencies to b4). Note that k=0 tok=3 since there is b0+b1+b2+b3, b4 (though k=1 is meaningless since thesubband SAD of b4 with itself is zero), b5, and b6+7.

In Eq. 21a, for FWD4, the closeness of b4 with b5 and b6+7 is evaluated,so the numbering for k goes from 1-3 (though should go from 2 to 3 since1 is meaningless as explained above). In Eq. 21b, numbering for k goesfrom 0 to 1, since only the zeroth sample is checked (i.e., the b0+1+2+3sample, and again, 1 is meaningless since b4 always matches itself).

Accordingly, using SubbandCountFWD4, SubbandCountBAK4, andSubbandCount8, TemporalMode is set as follows:

If SubbandCount8==4, then TemporalMode=MODE8;

-   -   else if SubbandCountFWD4==3 then TemporalMode=FWD4;    -   else if SubbandCountBAK4==2 then TemporalMode=BAK4;    -   else TemporalMode=SPATIAL.        Note that this scheme favors FWD4 over BAK4, but if MODE8 is not        signaled, then only one of FWD4 or BAK4 can be satisfied anyway.

Thresholding is performed on the 2D or 3D transformed blocks during theoverlapped block processing. For instance, when TemporalMode is set toSPATIAL, the 2D threshold module 306 is signaled, enabling the 2Dthreshold module 306 to perform 2D thresholding of the 2D transformedblock bs(0:63, 1) from F4(t). According to this mode, no thresholdingtakes place on the three (3) blocks from MSUM0123(t), M5(t) or MSUM67(t)(i.e., there is no 2D thresholding on bs(0:63, 0), bs(0:63, 2), bs(0:63,3)).

Reference is made to FIG. 10, which is a schematic diagram 1000 thatillustrates one example embodiment of time-space frequency partitioningby thresholding vector, T_(—)2D, and spatial index matrix, S_(—)2D, eachof which can be defined as follows:

T_(—)2D(0:3): 4-element vector of thresholds

S_(—)2D(0:3,2): 4×2 element matrix of spatial indices

Together, T_(—)2D and S_(—)2D define the parameters used forthresholding the 2D blocks bs(0:63, 1). For only the 8×8 2D transformedblock bs(0:63, 1), the thresholded block bst(0:63, 1) may be derivedaccording to Eq. (22) below:

$\begin{matrix}{{{bst}\left( {z,t} \right)} = \left\{ {{\begin{matrix}0 & {\begin{matrix}{{{if}\mspace{14mu} {{{bs}\left( {z,1} \right)}}} < {{T\_}2{D(j)}\mspace{14mu} {and}}} \\{{{S\_}2{D\left( {j,0} \right)}} \leq z \leq {{S\_}2{D\left( {j,1} \right)}}}\end{matrix}\mspace{14mu}} \\{{bs}\left( {z,1} \right)} & {otherwise}\end{matrix}{for}\mspace{14mu} j} = {0\mspace{14mu} \ldots \mspace{14mu} 3.}} \right.} & {{Eq}.\mspace{14mu} (22)}\end{matrix}$

In Eq. (22), T_(—)2D(j) defines the threshold used for block 1,bs(0:63, 1) from M4(t), over a span of spatial indices S_(—)2D(j, 0) toS_(—)2D(j, 1) for j=0.3. Equivalently stated, elements of bs(S_(—)2D(j,0): S_(—)2D(j, 1), 1) are thresholded by comparing the values of thoseelements to the threshold T_(—)2D(j), and the values in bs(S_(—)2D(j,0): S_(—)2D(j, 1), 1) are set to zero when their absolute values areless than T_(—)2D(j). Note that none of the matched frames MSUM0123(t),MSUM67(t) or M5(t) undergo 2D thresholding, only blocks from M4(t),bs(0:63, 1).

The spatial index matrix S_(—)2D together with T_(—)2D define a subsetof coefficients in the zig-zag scanned spatial frequencies asillustrated in FIG. 10. The 2D space has been reduced to 1D by zig zagscanning

Thresholding (at 3D threshold module 310) of the 3D transformed blocksbhaar(0:63, 0:3) output from 1D transform module 308 is performed whenTemporalMode is set to either FWD4, BAK4 or MODE8. Otherwise, whenTemporalMode is set to SPATIAL, the output of 3D thresholding module 310(e.g., bhaart(0:63, 0:3)) is set to the input of the 3D thresholdingmodule 310 (e.g., input equals bhaar(0:63, 0:3)) without modification.The 3D thresholding module 310 uses threshold vectors T_(—)3D(j) andspatial index matrix S_(—)3D(j, 0:1) defined hereinabove in 2Dthresholding 306, except using eight (8) thresholds so 0≦j<8.

An additional threshold vector TSUB(j, 0:1) is needed for 3Dthresholding 310, which defines the range of temporal subbands for eachj. For example, TSUB(0,0)=0 with TSUB(0, 1)=1 along with T_(—)3D(0)=100and S_(—)3D(0, 0)=0 and S_(—)3D(0, 1)=32 indicates that for j=0, 3Dthresholding with the threshold of 100 is used across Haar subbands L20and H20 and spatial frequencies 0 to 32.

Thresholding 310 of the 3D blocks is followed identically to the 2Dthresholding 306 case following Eq. (22), but substituting T_(—)3D andS_(—)3D for T_(—)2D and S_(—)2D. For 3D thresholding 310, unlike 2D, allfour (4) blocks bhaar(0:63, 0:3) are thresholded.

For MODE8, all Haar subbands are thresholded. For FWD4 and BAK4, only asubset of the Haar subbands is thresholded. The following specifieswhich subbands are thresholded according to TemporalMode:

If TemporalMode==MODE8 threshold [L20, H20, H02, H11] Haar Subbands;

If TemporalMode==FWD4 threshold [H20, H02, H11] Haar Subbands;

If TemporalMode==BAK4 threshold [L20, H20] Haar Subbands.

The temporal mode determinations described above are further facilitatedby another preprocessing sub-system embodiment, referred to herein asthe SCD sub-system 300B shown in FIGS. 3 and 5. In one embodiment, logicof the SCD sub-system 300 resides in the full pixel BM 516 as shown(FIG. 5), though one having ordinary skill in the art should appreciatethat other locales are contemplated for the SCD sub-system functionalityin some embodiments. Generally, scene change detection and handling areutilized for proper temporal mode decisions. During a scene change, forexample, even if 99% of the temporal mode decisions are fairly optimal,there exists a chance of a few bad decisions bleeding into the denoisedframe and becoming visible. In one embodiment, scene changes aredetected for the F7 frame (see, e.g., FIGS. 2C and 4) during FULLBM (oran F8 and delayed one frame). In general, all modes are reverted tobackward and spatial-only (no forward) during a scene change, andforward and spatial-only (no backward)—four (4) frames later—as the oldscene shifts into F0 to F3. When the scene change is completely flushed(F0 on previous frame), then mode decision processing reverts to normaloperation in the manner described above.

More specifically, scene change detection occurs during full pixel blockmatching (e.g., corresponding to full pixel BM 516), defined above inassociation with FIG. 5, for luma only (chroma reuses the luma scenechange detect, although in some embodiments, not limited to exclusivelyre-using luma). For each of the 8×8 luma blocks in full pixel blockmatching, if the final SAD in Eq. (14) is above a defined threshold,TSAD_SCENE_DETECT_THRESH, a counter is incremented for the frame. Whenthe counter exceeds another threshold,T_NUM_SAD_ABOVE_SCENE_DETECT_THRESH, a scene change detection istriggered. The T_NUM_SAD_ABOVE_SCENE_DETECT_THRESH parameter is set assome percentage of the total 8×8 blocks in the frame (e.g., 40%), thoughin some embodiments, other criteria may be used for the settingconfiguration. Although the discussion above is described in the contextof incrementing a counter and triggering of scene change detectionresponsive to “exceeding” certain thresholds, one having ordinary skillin the art should understand that other event triggering criteria (e.g.,greater than or equal to, etc.) may be employed in some embodiments.

In one embodiment, scene change detection is triggered only during theF7→F6 block matching, since this is the most distant frame later in timefrom F4. Also, since block matching proceeds for both fieldsindependently in interlaced video, a scene change detection is triggeredwhenever either of the fields detects a scene change. FIG. 11 shows adiagram 1100 of an index or table of matched frames 1102 (M0, M1, . . .M7) and a corresponding scene change detection mask (SCDM) 1104maintained by logic of the SCD sub-system 300B. When a scene change isdetected, a bit is set in the SCDM 1104. In this example embodiment, theSCDM 1104 comprises a mask which indicates the scene change detectionhistory for the current time and previous seven (7) frame times. Foreach frame processing interval, this mask 1104 is shifted to the leftone bit and a new bit is set on the right, denoting the scene changestatus (scene change detected) for the current M7. If matching F8→F7(and delaying, instead of F7→F6), then everything is delayed one framewith equivalent effect. It should be understood by one having ordinaryskill in the art in the context of the present disclosure that the useof eight (8) frames is only one example embodiment used for illustrativepurposes, and other quantities of frames may be used, enabling larger orsmaller masks than the mask 1104 shown in FIG. 11.

From the description above, it should be appreciated that the SCDsub-system 300B implements a scene change detection process that yieldsthe SCDM 1104, the SCDM 1104 indicating the scene change history forframes M0 through M7. The SCDM 1104 is communicated to the temporal modemodule 302. With a bit of the SCDM 1104 set for a given window offrames, the temporal mode decision described above in association withFIG. 9 is altered as given in the example pseudocode below:

Mode = GetTemporalMode  if Scene Change Mask Forward Bits Set  {   ifMode == BI8   {     Mode = BAK4   }   else if Mode == FWD4   {     Mode= SPATIAL   }  }  if Any Scene Change Mask Backward Bits Set  {   ifMode == BI8   {   Mode = FWD8   }   else if Mode == BAK4   {     Mode =SPATIAL   } }

Note that chroma processing uses the luma SCDM 1104 (e.g., the mask 1104is set once in the luma pixel BM process). Addressing the cooperationbetween the temporal mode determinations described above in associationwith FIG. 9 and the SCD sub-system 300B, one method embodiment comprisesfirst determining the temporal mode in the manner as described inassociation with FIG. 9. Then, if the SCDM 1104 has any forward bitsset, in accordance with the pseudocode hereinabove, the mode is revertedto backward only or spatial only modes. If the SCDM 1104 has anybackward bits set, then the mode is reverted to forward only or spatialmode. Note that both forward and backward bits may be set for very fastscene changes (e.g., less than eight (8) frames), during a continualcross-fade, etc. If no bits are set in the SCDM 1104, then the temporalmode as initially determined is left unaltered.

Having described example embodiments of thresholding in VDN systems,attention is again directed to FIG. 3 and inverse transform and outputprocessing. Specifically, inverse transforms include the inverse 1Dtransform 312 (e.g., Haar) followed by the inverse 2D transform (e.g.,AVC integer DCT), both described hereinabove. It should be understood byone having ordinary skill in the art that other types of transforms maybe used for both dimensions, or a mix of different types of transformsdifferent than the Haar/AVC integer combination described herein in someembodiments. For a 1D Haar transform, the inverse transform proceeds asshown in FIG. 8D and explained above.

The final denoised frame of FD4(t) using the merged accumulation buffersis specified in Eq. (23) as follows:

$\begin{matrix}{{{FD}\; 4\left( {i,j} \right)} = \frac{A\left( {i,j} \right)}{W\left( {i,j} \right)}} & {{Eq}.\mspace{14mu} (23)}\end{matrix}$

For uniform weighting where w(i,j)=1 for all overlapped blocks, Eq. (23)amounts to a simple divide by 16 (for step size=2). However, ifselectively omitting blocks, then Eq. (23) amounts to division by anumber 1≦W(i,j)≦16. After the 2D inverse transform 314 follows the 1Dinverse transform 312, the block bd(i,j,1) represents the denoised blockof F4(t). This denoised block is accumulated (added into) in the A4(t)accumulation buffer(s) 360 (e.g., accumulates the denoised estimates viarepeated loops back to the 2D transform module 304 to index or shift inpixels to repeat the processing 350 for a given reference frame), andthen output via normalization block 370.

Having described various preprocessing sub-systems of the example VDNsystems 200 disclosed herein, it should be appreciated that one methodembodiment 1200A, implemented by the field-frame processing sub-system300C in conjunction with an overlapped block processing module 350 andshown in FIG. 12A, comprises providing a field-frame map by partitioninga block matched frame into plural non-overlapping superblocks (1202);designating each of the plural superblocks of the field-frame map aseither field or frame (1204); comparing an overlapped superblock tofirst plural superblocks of the field-frame map (1206); and fieldprocessing by overlapped block processing logic two overlapped blocks ofthe overlapped superblock if one of the first plural blocks intersectedby the overlapped superblock has a field designation, otherwise frameprocessing, by the overlapped block processing logic, the two overlappedblocks of the overlapped superblock (1208).

Another method embodiment, 1200B, implemented by the field-frameprocessing sub-system 300C in conjunction with an overlapped blockprocessing module 350 and shown in FIG. 12B, comprises partitioning ablock matched reference frame into plural n×m non-overlapping pixelsuperblocks, where n and m are non-negative integer numbers (1210);designating each of the n×m pixel superblocks as field or frame (1212);and field processing by overlapped block processing logic two n×n blocksof an n×m overlapped superblock if one of first plural n×m blocksintersected by the overlapped superblock has a field designation,otherwise frame processing, by the overlapped block processing logic,the two n×n blocks of the overlapped superblock (1214).

Another method embodiment 1300, shown in FIG. 13, implemented by the SCDsub-system 300B, comprises receiving noise-filtered plural blocks of afirst frame and noise-filtered plural blocks of a second frame (1302);for each of the plural blocks to be matched, determining whether anindication of closeness in match between the each of the plural blocksexceeds a first threshold (1304); incrementing a counter value each timethe first threshold is exceeded for closeness of the block matching of aparticular block (1306); determining whether the counter value exceeds asecond threshold, the exceeding of the second threshold indicating thata defined quantity of blocks has exceeded the first threshold (1308);and responsive to determining that the counter value exceeds the secondthreshold, triggering a scene change detection (1310).

Another method embodiment 1400, shown in FIG. 14, implemented by the SMCsub-system 300A, comprises receiving at a frame matching module a firstframe comprising first plural blocks and plural frames that eachcomprise a plurality of blocks to be matched to the first plural blocksof the first frame, the first plural blocks and the plurality of blockscomprising luma blocks (1402); for each of the frame pair matchings,selecting one border configuration among a plurality of borderconfigurations, the border configuration selected for the each of theframe pair matchings unique (1404); appending a border of pixels to theframes of the each of the frame pair matchings based on the selectedborder configuration (1406); and block matching the first plural blockswith the plurality of blocks (1408).

Any process descriptions or blocks in flow charts or flow diagramsshould be understood as representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process, and alternateimplementations are included within the scope of the present disclosurein which functions may be executed out of order from that shown ordiscussed, including substantially concurrently or in reverse order,depending on the functionality involved, as would be understood by thosereasonably skilled in the art. In some embodiments, steps of a processidentified in FIGS. 12A-14 using separate boxes can be combined.Further, the various steps in the flow diagrams illustrated inconjunction with the present disclosure are not limited to thearchitectures described above in association with the description forthe flow diagram (as implemented in or by a particular module or logic)nor are the steps limited to the example embodiments described in thespecification and associated with the figures of the present disclosure.In some embodiments, one or more steps may be added to one or more ofthe methods described in FIGS. 12A-14, either in the beginning, end,and/or as intervening steps.

It should be emphasized that the above-described embodiments of thepresent disclosure are merely possible examples of implementations,merely set forth for a clear understanding of the principles of the VDNsystems and methods and their associated sub-systems. Many variationsand modifications may be made to the above-described embodiment(s)without departing substantially from the spirit and principles of thedisclosure. Although all such modifications and variations are intendedto be included herein within the scope of this disclosure and protectedby the following claims, the following claims are not necessarilylimited to the particular embodiments set out in the description.

1. A method, comprising: receiving at a frame matching module a firstframe comprising first plural blocks and plural frames that eachcomprise a plurality of blocks to be matched to the first plural blocksof the first frame, the first plural blocks and the plurality of blockscomprising luma blocks; for each of the frame pair matchings, selectingone border configuration among a plurality of border configurations, theborder configuration selected for the each of the frame pair matchingsunique; appending a border of pixels to the frames of the each of theframe pair matchings based on the selected border configuration; andblock matching the first plural blocks with the plurality of blocks. 2.The method of claim 1, further comprising removing the appended borderof pixels at a time corresponding to completion of the block matching.3. The method of claim 1, wherein the plurality of border configurationsdiffer from one another in number of pixels for at least one of theborders.
 4. The method of claim 1, wherein the effect of the appendedborder pixels according to the respective unique border configurationsresults in varying coordinate positions for seams among the first pluralblocks.
 5. The method of claim 4, further comprising providing blockmatched frames to a 3D video denoising module, wherein the effect of thevarying coordinate positions for the seams among the first plural blocksresults an amelioration of seam artifacts in corresponding observedvideo.
 6. The method of claim 1, wherein the plurality of borderconfigurations each comprises an even quantity of total border pixels.7. The method of claim 1, further comprising receiving chroma versionsof the first and the plural frames associated with the each of the framepair matchings and appending to the chroma versions of the frames adownsampled version of the unique configuration of border pixelsaccording to the selected border configuration for the each of the framepair matchings.
 8. The method of claim 1, wherein for interlaced video,each field of the first frame comprises a different borderconfiguration.
 9. The method of claim 1, wherein the appended border ofpixels comprise black pixels.
 10. A system, comprising: a frame matchingmodule comprising border add logic, the border add logic configured to:receive a first frame comprising first plural blocks and plural framesthat each comprise a plurality of blocks to be matched to the firstplural blocks of the first frame, the first plural blocks and theplurality of blocks comprising luma blocks; for each of the frame pairmatchings, select one border configuration among a plurality of borderconfigurations, the border configuration selected for the each of theframe pair matchings unique; append a border of pixels to the frames ofthe each of the frame pair matchings based on the selected borderconfiguration; and motion estimation (ME) and motion compensation (MC)logic of the frame matching module, the ME and MC logic configured to:block match the first plural blocks with the plurality of blocks. 11.The system of claim 10, further comprising border extract logicconfigured to remove the appended border of pixels at a timecorresponding to completion of the block matching.
 12. The system ofclaim 10, wherein the plurality of border configurations differ from oneanother in number of pixels for at least one of the borders.
 13. Thesystem of claim 10, wherein the effect of the appended border pixelsaccording to the respective unique border configurations results invarying coordinate positions for seams among the first plural blocks.14. The system of claim 13, further comprising a 3D video denoisingmodule configured to received the block matched frames post-pixel borderremoval, the 3D video denoising module configured to provide for outputthe denoised first frame with the varying coordinate positions for theseams among the first plural blocks resulting in an amelioration of seamartifacts in corresponding observed video.
 15. The system of claim 10,wherein the plurality of border configurations each comprises an evenquantity of total border pixels.
 16. The system of claim 10, furthercomprising second border add logic corresponding to chroma processing,the second border add logic configured to receive chroma versions of thefirst and the plural frames associated with the each of the frame pairmatchings and append to the chroma versions of the frames a downsampledversion of the unique configuration of border of pixels according to theselected border configuration for the each of the frame pair matchings.17. The system of claim 10, wherein for interlaced video, each field ofthe first frame comprises a different border configuration.
 18. Thesystem of claim 10, wherein the appended border of pixels comprise blackpixels.
 19. A staggered motion compensated (SMC) pre-processingsub-system, comprising: border add logic configured to: receive a firstframe comprising first plural blocks and plural frames that eachcomprise a plurality of blocks to be matched to the first plural blocksof the first frame, the first plural blocks and the plurality of blockscomprising luma blocks; for each of the frame pair matchings, select oneborder configuration among a plurality of border configurations, theborder configuration selected for the each of the frame pair matchingsunique; append a border of pixels to the frames of the each of the framepair matchings based on the selected border configuration; and providethe frames with the appended pixels to block matching logic.
 20. The SMCpre-processing sub-system of claim 19, wherein the effect of theappended border pixels according to the respective unique borderconfigurations results in varying coordinate positions for seams amongthe first plural blocks.